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CHAPTER I. HISTORY AND OVERVIEW OF THE 
NATIONAL REPORTING SYSTEM 


The National Reporting System (NRS) is the accountability system for the federally funded, 
State-administered adult education program. It addresses the accountability requirements of the Adult 
Education and Family Literacy Act, Title II of the Workforce Investment Act (WIA—P.L.105—220). 
This document presents (1) the NRS measures that allow assessment of the impact of adult education 
instruction, (2) methodologies for collecting the measures, (3) reporting forms and procedures, and 
(4) training and technical assistance requirements to assist States in collecting and reporting the 
measures. 


HISTORY OF THE NRS 


The NRS was born in the 1990s, a decade known for its emphasis on accountability of 
Federal programs, when all publicly funded programs and agencies faced increasing pressures to 
demonstrate that they had met their legislative goals and had an impact on their client populations. 
The requirement to demonstrate program impact was mandated in 1993 through the Government 
Performance and Review Act (GPRA). GPRA required all Federal agencies to develop strategic 
plans to ensure that services were delivered efficiently and in a manner that best suited client needs 
and to develop indicators of performance to demonstrate their agency’s impact. 


In 1995, the U.S. Congress considered eliminating adult education as a separate delivery 
system by integrating the program into a general system of workforce development. Strong and 
convincing data on the impact of adult education at the State and Federal levels were demanded to 
demonstrate its importance as a separate education program. In response to these demands, the State 
directors of adult education asked the Division of Adult Education and Literacy (DAEL) to work 
toward developing a national system for collecting information on adult education student outcomes. 


To address these demands, DAEL devoted its March 1996 national meeting of State directors 
of adult education to developing a framework for program accountability. This framework specified 
the purposes of the adult education program and the essential characteristics of an accountability 
system and identified seven categories of outcome measures. At the March 1997 DAEL national 
meeting, a broad group of adult education stakeholders validated the framework, identified outcome 
measures for a new national reporting system, and discussed possible methodologies for the system. 
Based on these decisions, a project to design and develop the reporting system began in October 
1997. The proposed voluntary nature of the NRS changed in August 1998 with the passage of WIA, 
which required an accountability system. The NRS mandate was then expanded to establish the 
measures and methods to conform to WIA requirements. 


NRS Development Phases 


The goals of the NRS project were to develop a national accountability system for adult 
education programs by identifying measures for national reporting and their definitions, establishing 
methodologies for data collection, developing standards for reporting to the U.S. Department of 
Education, and developing training materials and activities on NRS requirements and procedures. 
The development of the NRS proceeded in three phases. 
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The first phase, standardization, involved the development of standard measure definitions 
for State and local programs, standard data collection methodologies, and software standards for 
automated data reporting. In the summer of 1998, interim software standards were established, 
methodologies were identified for pilot testing, and draft definitions for use in the pilot test were 
distributed to adult education stakeholders. 


The pilot test was the second phase of the project and was designed to have a small number 
of volunteer States and local programs test the draft measure definitions and proposed methodologies 
under realistic conditions. The pilot test assessed whether the draft measure definitions worked or 
needed to be refined. It also assessed costs, burden, and other difficulties in collecting the data using 
the proposed methodologies. The pilot test was completed in January 1999. Measures and 
methodologies were revised based on the pilot test. 


The third phase of the project, training and technical assistance, which began in the summer 
of 2003, supported State and local program implementation of the NRS. The different types of 
assistance included instructional training packets for States to use in a “train the trainer” 
environment, technology-based materials for State and local staff that explained NRS measures and 
methods, and individual technical assistance to States that supported their implementation efforts. 


An advisory board—consisting of State directors of adult education, representatives from 
volunteer provider agencies, directors of local adult education programs, and experts on 
accountability systems—guided the project and met three times between December 1997 and March 
1999. The board made significant substantive contributions to the measure definitions and 
methodologies. Participants in the pilot test also provided advice and guidance on measures and 
methods. 


DAEL released a draft of the NRS Implementation Guidelines in mid-1999 and another draft 
in June 2000, reflecting changes from State comments and early State experiences in implementing 
the requirements. The NRS formally went into effect on July 1, 2000, and DAEL issued a final 
Guidelines document in March 2001. Since that time there have been periodic updates to the 
Guidelines to clarify methodologies, provide further guidance on data collection and revisions to 
measures in response to ED requirements. 


Revisions 


e The Office of Career, Technical, and Adult Education (OCTAE) published proposed 
modifications to the learning objectives associated with each educational functioning level (EFL) 
identified for Adult Basic Education (ABE) in the NRS and requested public comments in the 
Federal Register notice ED-2015-ICCD-0004-0001. A summary of the public comments 
received and OCTAE’s responses are included in Appendix D. The new EFL descriptors for 
ABE are included in Appendix E. The new descriptors will not be implemented until the 
Secretary has determined that there is at least one assessment that is both aligned with the revised 
descriptors and that is suitable for use in the NRS. Until that time, programs will continue to use 
the existing NRS educational functioning level descriptors in Exhibit 2.1 on pages 16-21. 


e The Narrative Report instructions have been streamlined and updated. Two new prompts ask 
grantees to report the progress they are making on important initiatives. 


e All references to specific secondary school credentials or high school equivalency tests have been 
revised. 
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OVERVIEW OF THE NRS MEASURES AND METHODS 


NRS Measures 


The requirements of WIA, consensus among the stakeholders and advisory board members, 
and the need for uniform valid and reliable data were major factors guiding the development of NRS 
measures. Other factors affecting the development of the measures included the need to 
accommodate the diversity of the adult education delivery system and the need for compatible 
definitions between related adult education and training programs. 


As a State-administered program, the nature of adult education service delivery varies widely 
across States in its goals, objectives, and the resources available to States to collect and report data. 
It is especially important that the definitions for outcome measures be broad enough to accommodate 
these differences, yet concrete and standardized sufficiently to allow the NRS to establish a uniform 
national database. 


To accommodate the diverse delivery system and compatibility with related systems, NRS 
staff conducted a thorough review of measure definitions planned or in use by all States and all 
Federal employment and training programs. To identify State measures used, for example, NRS staff 
conducted an evaluability assessment of all States in early 1998 and obtained copies of measure 
definitions from States that had their own measures. In addition, NRS staff reviewed the existing 
measure definitions used for DAEL’s annual statistical performance report and measures and 
definitions planned by the U.S. Department of Education for Title I of WIA. A full list of the main 
sources consulted in developing the measures and definitions is provided at the end of this chapter. 


Exhibit 1.1 lists the core and secondary measures of the NRS. The core measures apply to 
all adult education students receiving 12 or more hours of service. There are three types of core 
measures: 


e Outcome measures include educational gain, entered employment, retained 
employment, receipt of secondary school credential or its equivalent, and placement in 
postsecondary education or training. 
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Exhibit 1.1 
Summary of NRS Measures and Definitions 


Topic Measures Categories or Definitions 


Core Outcome Measures 


Educational gains “* Educational gains «* Educational functioning levels in 
reading, writing, speaking, and 
listening and functional areas 


Follow-up measures Entered employment Learners who obtain a job by the 
end of the first quarter after the exit 
quarter 


Retained employment Learners who obtain a job and 
remain employed in the third 
quarter after program exit 


Secondary school credential or its Learners who obtain a secondary 
equivalent school diploma, or recognized 
equivalent after exit. 


Placement in postsecondary Learners enrolling after exit in a 

education or training postsecondary educational or 
occupational skills program building 
on prior services or training 
received 


Core Descriptive and Participation Measures 


Demographics “* Race/Ethnicity «* American Indian or Alaska Native, 
Asian, Native Hawaiian or Other 
Pacific Islander, Black or African 
American (non-Hispanic), Hispanic 
or Latino, White (non-Hispanic) 


“+ Gender “+ Male, female 
“* Age “+ Date of birth 
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Exhibit 1.1 (Continued) 
Summary of NRS Measures and Definitions 


Topic Measures Categories or Definitions 


Core Descriptive and Participation Measures 


«* Labor force status «* Employed, not employed, not in 
labor force 


7 


** Public assistance status Receiving or not receiving 
assistance 


Rural residency Rural, not rural 


Disability status Disabled, not disabled 


Highest degree or level of school Highest grade level of school 

completed completed in US or abroad, 
college, secondary credential, 
postsecondary degree 


Student participation Contact hours Number of hours of instructional 
activity 


Program enrollment type ABE, ASE, ESL, family literacy, 
workplace programs, homeless 
programs, correctional facilities, 
community corrections programs, 
other institutional programs 


Teacher status Total years of experience Total number of years of 
experience teaching 


Certification in K-12, special 
education, adult education, TESOL 


Secondary Outcome and Student Status Measures (Optional) 


Employment “* Reduction in receipt of public «* Students whose welfare benefits 
assistance or equivalent public assistance 
grant is reduced or eliminated due 
to employment 


“¢ Teacher certification 


\2 


Work-based project learner *¢ Met work-based project learner ** Achieved skills for work-based 

achievement goal project learner activity (activity of at 
least 12 hours and no more than 30 
hours of instruction related to 
specific workplace skills) 
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Exhibit 1.1 (Continued) 
Summary of NRS Measures and Definitions 


Topic Measures Categories or Definitions 


Secondary Outcome and Student Status Measures (Optional) 


Community ** Achieved citizenship skills ** Achieve the skills needed to pass 
the citizenship exam 


Voting behavior Learner registers to vote or votes 
for the first time 


General involvement in community Learner increases involvement in 
activities community activities 


o, 


Involvement in children’s education + Learner increases help given for 
children’s school work, contact with 
teachers to discuss education, and 
involvement in children’s school 


Involvement in children’s literacy- Learner increases the amount read 

related activities to children, visits libraries, or 
purchases books or magazines for 
children 


Student status Low-income status Low income, not low income 


Displaced homemaker Displaced homemaker, not 
displaced homemaker 


Single-parent status Single parent, not single parent 


Dislocated worker Dislocated worker, not dislocated 
worker 


Learning disabled adult Learning disabled, not learning 
disabled 


Learner's goals for attending Obtain a job, retain current job, 
improve current job, earn a 
secondary school credential or its 
equivalent, enter postsecondary 
education or training, improve 
basic literacy skills, improve 
English language skills, 
citizenship, work-based project 
learner goal, other personal goals 


e Descriptive measures include student demographics, reasons for attending, and student 
status. 


e Participation measures include contact hours received and enrollment in instructional 
programs for special populations or topics, such as family literacy or workplace literacy. 


The NRS secondary measures include additional outcome measures related to employment, 
family, and community. Adult education stakeholders believe these are important to understanding 
and evaluating adult education programs. States are not required to report on the secondary 
measures and no performance standards are tied to them. The optional secondary measures are not 
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used as a basis for incentive grant awards. There also are secondary student status measures that 
define target populations identified in WIA. These measures are provided for States that want to 
report on the services provided to these populations. 


Core Outcome Measures 


Student outcome measures are the central measures of the NRS. Although they are not the 
only measures that could be used to evaluate adult education programs, the outcome measures 
selected represent what a broad consensus of adult educators believe are appropriate measures for 
providing a national picture of program performance. The multiyear process employed by the NRS to 
identify and define the measures included input from State directors of adult education, Federal 
education officials, local education providers, representatives of volunteer literacy organizations, and 
experts in performance accountability systems. 


The five NRS core outcome measures were selected to address the requirements for core 
indicators of performance in the Adult Education and Family Literacy Act of the WIA. Exhibit 1.2 
shows how the measures relate to these requirements and goals for adult education stated in the 
legislation. 


Educational gain, a key outcome in the NRS, provides a measure of student literacy gains 
resulting from instruction. This measure applies to all students in the program (except pre-designated 
“work-based project learners,” which is described below under “Secondary Measures”). To 
determine this measure, local programs assess students on intake to determine their educational 
functioning level. There are four levels for adult basic education (ABE), two for adult secondary 
education (ASE), and six levels of ESL. Each level describes a set of skills and competencies that 
students entering at that level can do in the areas of reading, writing, numeracy, speaking, listening, 
and functional and workplace areas. Using these descriptors as guidelines, programs determine the 
appropriate initial level at which to place students using a standardized assessment procedure (i.e., a 
test or a standardized performance-based assessment). The program decides the skill areas in which 
to assess the student based on the student’s instructional needs. 


Exhibit 1.2 
Goals and Core Indicators of the WIA Adult Education and 
Family Literacy Act and NRS Core Outcome Measures 


Goals of Adult Education 
Described in the Adult Education Core Indicators Required by the 
and Family Adult Education and Family 
Literacy Act of WIA Literacy Act of WIA NRS Core Outcome Measures 

Assist adults to become literate and Improvements in literacy skill levels in Educational gains (achieve skills 

obtain the knowledge and skills reading, writing, and speaking the to advance educational 

necessary for employment and self- English language; numeracy; problem | — functioning level) 

sufficiency solving; English-language acquisition; 
other literacy skills 

Assist parents to obtain the skills Placement in, retention in, or Entered employment 

necessary to be full partners in their completion of postsecondary Retained employment 

children’s educational development education, training, unsubsidized Placement in postsecondary 
employment, or career advancement education or training 


Assist adults in the completion of Receipt of a secondary school Receipt of a secondary school 
secondary school education diploma or its recognized equivalent credential or its equivalent 
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After a predetermined amount of instruction or time period determined by each State, the 
program conducts follow-up assessments of students in the same skill areas and uses the test scores 
aligned to the educational functioning levels to determine whether the students have advanced one or 
more levels or are progressing within the same level. The State has discretion to establish the 
standardized student assessment method used within the State, as well as procedures for progress 
assessment, and must develop a written statewide assessment policy describing assessments and 
procedures for approval from DAEL. All assessments and procedures must conform to standard 
psychometric criteria for validity and reliability as defined by DAEL. Upon DAEL approval, States 
may also use additional educational levels and skill area descriptors, as long as they are compatible 
with NRS levels and skills. 


The remaining core outcome measures are follow-up measures that are reported some time 
after the student exits the program. The follow-up measures apply to students who exit and who are 
members of the cohort specific to that measure (see definitions in Chapter II). For students who are 
not employed but in the labor force, there are two follow-up measures: entered employment (whether 
the student obtained a job by the end of the first quarter after leaving) and retained employment 
(whether students who entered employment still have a job in the third quarter after exit). The 
retained employment measure also applies to students who enter the program employed. For the 
measure of entry into further education or training, the applicable cohort includes all students who 
enter with a secondary credential, or who earned a secondary school credential or its equivalent while 
enrolled or who enrolled in a class designed for postsecondary transition. The measure of obtaining a 
secondary credential applies to enrolled students who take all parts of a State recognized high school 
equivalency test, or are enrolled in adult high school at the high ASE level, or are enrolled in the 
assessment phase of the External Diploma Program (EDP). The secondary credential cohort is 
defined by entry level EFL and only students who have exited the program are included in the 
cohort. 


Descriptive and Participation Measures 


The NRS descriptive measures are student demographics and status in several areas. These 
measures allow for a description and understanding of who attends adult education programs and for 
what reasons. The measures also allow for analyses of the performance of specific groups of students 
attending adult education programs, such as unemployed students or students receiving public 
assistance. The demographic measures include ethnicity, age, and gender; and status measures 
include employment status, highest degree achieved or level of schooling, and whether the student 
has a disability or is on public assistance. Teacher status measures include total years of experience 
in adult education and certifications. 


Two participation measures—contact hours and program enrollment type—are collected for 
both descriptive and analytic purposes. These measures record the amount of instruction students 
receive and the number of students attending in areas such as family literacy and workplace literacy. 


Secondary Measures 


The NRS secondary measures are optional measures of student outcomes and status that 
States are not required to use and should not be used as a basis for assessing State performance 
under WIA. No performance standards will be tied to these measures and they will not be used to 
determine State eligibility for incentive awards under WIA. The NRS includes these measures 
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because many stakeholders during the consensus-building process believed these measures were 
important to the identity of the program and the goals and purposes of adult education. 


The secondary measures are in the areas of employment, community, and family. The 
employment measure indicates whether the student’s public assistance grant was reduced or 
eliminated due to employment. This measure applies only to students receiving public assistance 
upon entry. 


In the area of community, there are three measures covering citizenship, voting, and 
community involvement. For students enrolled in EL Civics and citizenship programs, there is a 
measure of whether students have achieved citizenship skills. Voting for the first time or registering 
to vote and more involvement in community groups or activities are the remaining measures. The 
family measures include increased involvement in children’s literacy activities and in children’s 
education. 


A measure added to the NRS in 2000 is completed work-based project learner activity. 
Project learners are students enrolled in a class with 30 hours or less of scheduled instruction that has 
a goal of teaching specific workplace-related literacy skills. On enrollment, the learner and the 
program determine the specific skills to be learned and the method to assess the skill attainment. The 
assessment must employ a standardized test or be a performance-based assessment with standardized 
scoring rubrics. The assessment must conform to commonly accepted psychometric criteria for 
validity and reliability and meet standards for acceptable assessments, as defined by DAEL. 
Programs do not collect the core outcome measures on students designated as project learners, and 
these learners are counted separately. This measure is included within the NRS to allow States and 
programs to serve learners with a short-term learning need, without having a detrimental effect on 
core outcome measure performance. 


Secondary student status measures of low-income status, displaced homemaker, and single- 
parent status are included, because these groups are specific target populations under WIA. States 
that must report their services to these populations can use these measures, which are defined 
identically to the U.S. Department of Labor definitions. There is also a secondary status measure to 
identify learning-disabled adults to assist programs in reaching these students and a measure of 
learner’s goals for attending. 


NRS Methodologies 


To help ensure comparability of measures across States, the NRS has established procedures 
for collecting all of the NRS measures. The NRS has three methodologies for collecting measures: 
direct program reporting, local follow-up survey, and data matching. With the direct program 
reporting methodology, local programs collect the information directly from the learner while the 
learner is enrolled and receiving instruction. The information is normally obtained as part of the 
intake process (through student assessment) or ongoing throughout the course of instruction. 
Measures collected with this methodology are the demographic, student status, and student 
participation measures, as well as the educational gain measure and the secondary measures of 
project learner completion and citizenship skill attainment. 


Two methodologies, a follow-up survey or data matching, are offered for collecting the NRS 
core outcome measures that require follow-up—the employment-related measures, receipt of a 
secondary school credential or its equivalent, and placement in postsecondary education or training. 
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Follow-up methodologies also may be used to collect the optional secondary outcome measures. The 
local survey methodology employs a survey of learners who left the program during the program 
year. The local program, State, or third-party contractor may conduct the survey. To conduct this 
survey, programs must include all of the students in the program to whom the measure applies (see 
definitions) or, in some cases, programs can draw from a statistically valid random sample of learners 
from the cohort for whom the measure applies. The procedures for conducting the survey are to be 
determined by the State but must follow accepted scientific practice for producing valid results. The 
State is required to establish a policy for follow-up for DAEL approval that clearly describes the 
procedures to be followed. The entered employment measure must be collected during the first 
quarter after the student exits the program. Retained employment must be collected in the third 
quarter after exit, and the other measures can be collected at any time during the year after the 
student exits the program. 


An alternative way to collect the follow-up measures is a data matching methodology. Data 
matching refers to the procedures whereby agencies serving common clients pool their data to 
identify outcomes unique to each program. Matching is achieved using student Social Security 
numbers (or other unique identifiers) and is typically done at the State level. For example, to 
determine whether students obtained employment after leaving the program, the State adult education 
agency would match the Social Security numbers and dates of attendance of students who had 
obtained employment in the State wage record database for the appropriate calendar quarter. States 
may use either follow-up method, or a combination of the two methods, to collect NRS follow-up 
measures. 


NRS Guidebooks, Resources, and Revised Guidelines 


Since the NRS was implemented, DAEL has offered additional training and guidebooks to 
clarify NRS requirements and to assist States in the collection and use of quality data for program 
management and improvement efforts. The following guidebooks were prepared by NRS project 
staff at the American Institutes for Research (AIR): 


e The Guide for Improving NRS Data Quality explains in greater detail ways to standardize and 
improve data collection procedures for the NRS. 


e Using NRS Data for Program Management and Improvement offers a data use and program 
change model and suggests ways to use NRS data. 


e NRS Data Monitoring for Program Improvement explains the use of performance standards 
in program monitoring and suggests ways for States to effectively monitor local program 
performance. 


e Developing an NRS Data System offers help to State and local adult education staff so that 
they can make informed decisions on the design and development of a data system for the 
NRS. It outlines a process for identifying requirements that reflect the range of needs from 
functional and operational perspectives. 


e The Third Wave of the NRS provides a review of the status of the accountability system to 
date and previews changes and policies being considered to improve the NRS. 
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e Demonstrating Results: Developing State and Local Report Cards for Adult Education 
explains the components of report cards for demonstrating state and program performance, 
how they are used for program improvement and how to develop them. 


e Desk Monitoring: Improving Program Performance focuses on developing a desk monitoring 
system, including a tool to supplement onsite monitoring visits and a rubric to evaluate 
program performance. 


e Learning to be an NRS Data Detective: The Five Sides of the NRS addresses several aspects 
of the NRS including data collection procedures and requirements, improving data quality, 
and using data for the NRS. 


e Building and Sustaining Quality in NRS Data: Strategies for Program Improvement: dealt 
with using NRS data to improve program quality through a four step continuous 
improvement process for building and sustaining change. 


e The Local Connection: Building a Data Use Learning Community focused on the 
characteristics and essential elements of data use learning communities at the state and local 
level. 


e Smart Fiscal Management for Adult Education Programs organized and explained in ever 
day language, the basic financial requirements for managing the Federal adult education and 
literacy program. 


The guidebooks have accompanying training materials, and State staff was trained in the use 
of the guides and materials at regional trainings shortly after the release of each guidebook. 


The NRS support project Web site, NRSWEB (http://www.nrsweb.org), includes 
comprehensive information about the NRS, links to NRS documents and other resources, and online 
courses about NRS requirements, data quality, and data use for State and local adult education 
program staff. AIR has developed all NRS documents, training courses, and Web sites under contract 
to the U.S. Department of Education. 
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OVERVIEW OF THIS DOCUMENT 


The remainder of this document presents NRS measures, methods, reporting requirements, 
and NRS data collection policies in greater detail. Chapter II presents definitions of all NRS 
measures and the methodologies for collecting them. Chapter III presents an overview of the NRS 
data collection framework and describes how information flows from the classroom and program to 
the State and Federal levels. Chapter III discusses the responsibilities of Federal, State, and local 
agencies in the data collection and reporting processes. Chapter IV discusses quality control 
procedures and recommendations for local student record systems to enable NRS reporting. There 
are three appendixes. Appendix A offers a sample follow-up survey and model methodologies, 
appendix B presents a copy of the Data Quality Standards Checklist, and appendix C provides the 
NRS reporting tables. 
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The NRS includes core measures and secondary measures. The core measures are required 
and include outcome, descriptive, and participation measures that reflect the core indicator 
requirements of the WIA. States must report the required measures on all students who receive 12 
hours or more of service. For PY 2013, the U.S. Department of Education (ED) will only use the 
core measures for educational gain to judge State performance, including eligibility for incentive 
grants. ED will use PY 2013 data for the core follow-up outcome measures to set new baselines for 
future performance targets. The secondary measures include additional, optional outcomes related to 
employment, family, and community that adult education stakeholders believe are important to 
understanding and evaluating adult education programs. States are not required to report on the 
secondary measures and no performance standards are tied to them. The optional secondary 
measures are not used as a basis for incentive grant awards. There also are secondary measures of 
student status that include target populations identified under WIA. These measures are included for 
States that want to report on services provided to these populations. The definitions are identical to 
those used by the U.S. Department of Labor, which aids in uniform reporting under both Title I and 
Title II of WIA. 


This chapter presents the definitions for the all of the NRS measures, the applicable student 
population to which the measure applies, and Federal reporting requirements. Along with each 
measure is a discussion of the data collection policies and procedures that States and local programs 
should have in place to collect the measures. The chapter first presents the core outcome measures, 
followed by the required demographic, status, and participation measures. The chapter concludes 
with definitions and requirements for NRS secondary, optional measures. 


CORE OUTCOME MEASURES 


The NRS core outcome measures are: educational gain, entered and retained employment, 
receipt of a secondary credential, and entered postsecondary education. States set performance 
standards for these measures, and program effectiveness is judged in part by whether these standards 
are met. This section presents the definition, requirements, and methodology for each of these core 
measures. 


Educational Gain 


Educational gain measures the primary purpose of the adult basic education program: to improve the 
basic literacy skills of participants. This goal is the reason that all students are counted in the 
educational gain measure. The NRS approach to measuring educational gain is to define a set of 
educational functioning levels at which students are initially placed based on their abilities to 
perform literacy-related tasks in specific content areas. After a set time period or number of 
instructional hours set by the State, students are again assessed to determine their skill levels. If their 
skills have improved sufficiently to be placed one or more levels higher, an “advance” is recorded for 
that student. States that offer adult high school credit programs, (including adult high schools) may 
measure and report educational gain through the awarding of credits or Carnegie units. 


Exhibit 2.1 describes the educational functioning levels. 
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Exhibit 2.1 
Functioning Level Table 


Outcome Measures Definitions 


EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS— ADULT Basic EDUCATION LEVELS 


Beginning ABE Literacy Individual has no or minimal reading and writing skills. May have Individual has little or no recognition of Individual has little or no ability to read basic signs or maps and 
little or no comprehension of how print corresponds to spoken numbers or simple counting skills or may have | can provide limited personal information on simple forms. The 
Test Benchmark: language and may have difficulty using a writing instrument. At the | only minimal skills, such as the ability to add or | individual can handle routine entry level jobs that require little or 
TABE (9-10) scale scores upper range of this level, individual can recognize, read, and write | subtract single digit numbers. no basic written communication or computational skills and no 
(grade level 0-1.9): letters and numbers but has a limited understanding of connected knowledge of computers or other technology. 
Reading: 367 and below prose and may need frequent re-reading. Can write a limited 
Total Math: 313 and below number of basic sight words and familiar words and phrases; may 
Language: 389 and below also be able to write simple sentences or phrases, including very 
simple messages. Can write basic personal information. Narrative 
CASAS scale scores: writing is disorganized and unclear, inconsistently uses simple 
Reading: 200 and below punctuation (e.g., periods, commas, question marks), and contains 
Math: 200 and below frequent errors in spelling. 


Wonderlic GAIN scale scores: 
English: 200-406 
Math: 200-314 


Beginning Basic Education Individual can read simple material on familiar subjects and Individual can count, add, and subtract three Individual is able to read simple directions, signs, and maps, fill out 
comprehend simple and compound sentences in single or linked digit numbers, can perform multiplication simple forms requiring basic personal information, write phone 
Test Benchmark: paragraphs containing a familiar vocabulary; can write simple notes | through 12, can identify simple fractions, and messages, and make simple changes. There is minimal 
TABE (9-10) scale scores and messages on familiar situations but lacks clarity and focus. perform other simple arithmetic operations. knowledge of and experience with using computers and related 
(grade level 2-3.9): Sentence structure lacks variety, but individual shows some control technology. The individual can handle basic entry level jobs that 
Reading: 368-460 of basic grammar (e.g., present and past tense) and consistent use require minimal literacy skills; can recognize very short, explicit, 
Total Math: 314-441 of punctuation (e.g., periods, capitalization). pictorial texts (e.g., understands logos related to worker safety 
Language: 390-490 before using a piece of machinery); and can read want ads and 
complete simple job applications. 
CASAS scale scores: 
Reading: 201-210 
Math: 201-210 


Wonderlic GAIN scale scores: 
English: 407-525 
Math: 315-522 


MAPT scale scores: 
All tests: 200-299 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 
CASAS = Comprehensive Adult Student Assessment System e TABE = Test of Adult Basic Education e MAPT= Massachusetts Adult Proficiency Tests e Wonderlic GAIN = Wonderlic General Assessment of Instructional Needs 
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Exhibit 2.1 (Continued) 
Functioning Level Table 


Low Intermediate Basic Education 


Test Benchmark: 

TABE (9-10) scale scores 

(grade level 4-5.9): 
Reading: 461-517 
Total Math: 442-505 
Language: 491-523 


CASAS scale scores: 
Reading: 211-220 
Math: 211-220 


Wonderlic GAIN scale scores: 
English: 526-661 
Math: 523-669 


MAPT scale scores: 
All tests: 300-399 


High Intermediate Basic Education 
Test Benchmark: 
TABE (9-10) scale scores 
(grade level 6-8.9): 
Reading: 518-566 
Total Math: 506-565 
Language: 524-559 


CASAS scale scores: 
Reading: 221-235 
Math: 221-235 


Wonderlic GAIN scale scores: 
English: 662-746 
Math: 670-775 


MAPT scale scores: 
All tests: 400-499 


Outcome Measures Definitions 
EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS— ADULT Basic EDUCATION LEVELS 


Basic Reading and Writing Numeracy Skills Functional and Workplace Skills 


Individual can read text on familiar subjects that have a simple and 
clear underlying structure (e.g., clear main idea, chronological 
order); can use context to determine meaning; can interpret 
actions required in specific written directions; can write simple 
paragraphs with a main idea and supporting details on familiar 
topics (e.g., daily activities, personal issues) by recombining 
learned vocabulary and structures; and can self and peer edit for 
spelling and punctuation errors. 


Individual is able to read simple descriptions and narratives on 
familiar subjects or from which new vocabulary can be determined 
by context and can make some minimal inferences about familiar 
texts and compare and contrast information from such texts but not 
consistently. The individual can write simple narrative descriptions 
and short essays on familiar topics and has consistent use of basic 
punctuation but makes grammatical errors with complex 
structures. 


Individual can perform with high accuracy all 


four basic math operations using whole 


numbers up to three digits and can identify 


and use all basic mathematical symbols. 


Individual can perform all four basic math 
operations with whole numbers and fractions; 
can determine correct math operations for 
solving narrative math problems and can 
convert fractions to decimals and decimals to 
fractions; and can perform basic operations 


on fractions. 


Individual is able to handle basic reading, writing, and 
computational tasks related to life roles, such as completing 
medical forms, order forms, or job applications; and can read 
simple charts, graphs, labels, and payroll stubs and simple 
authentic material if familiar with the topic. The individual can use 
simple computer programs and perform a sequence of routine 
tasks given direction using technology (e.g., fax machine, 
computer operation). The individual can qualify for entry level jobs 
that require following basic written instructions and diagrams with 
assistance, such as oral clarification; can write a short report or 
message to fellow workers; and can read simple dials and scales 
and take routine measurements. 


Individual is able to handle basic life skills tasks such as graphs, 
charts, and labels and can follow multistep diagrams; can read 
authentic materials on familiar topics, such as simple employee 
handbooks and payroll stubs; can complete forms such as a job 
application and reconcile a bank statement. Can handle jobs that 
involve following simple written instructions and diagrams; can 
read procedural texts, where the information is supported by 
diagrams, to remedy a problem, such as locating a problem with a 
machine or carrying out repairs using a repair manual. The 
individual can learn or work with most basic computer software, 
such as using a word processor to produce own texts, and can 
follow simple instructions for using technology. 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 
CASAS = Comprehensive Adult Student Assessment System e TABE = Test of Adult Basic Education e MAPT: Massachusetts Adult Proficiency Tests ¢ Wonderlic GAIN = Wonderlic General Assessment of Instructional Needs 
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Exhibit 2.1 (Continued) 
Functioning Level Table 


Low Adult Secondary Education 


Test Benchmark: 

TABE (9-10): scale scores 

(grade level 9-10.9): 
Reading: 567-595 
Total Math: 566-594 
Language: 560-585 


CASAS scale scores: 
Reading: 236-245 
Math: 236-245 


Wonderlic GAIN scale scores: 
English: 747-870 
Math: 776-854 


MAPT scale scores: 
All tests: 500-599 


High Adult Secondary Education 


Test Benchmark: 

TABE (9-10): scale scores 

(grade level 11-12): 
Reading: 596 and above 
Total Math: 595 and above 
Language: 586 and above 


CASAS scale scores: 
Reading: 246 and above 
Math: 246 and above 


Wonderlic GAIN scale scores: 
English: 871-1000 
Math: 855-1000 


MAPT scale scores: 
All tests: 600-700 


Outcome Measures Definitions 
EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS— ADULT SECONDARY EDUCATION LEVELS 


Basic Reading and Writing Numeracy Skills Functional and Workplace Skills 


Individual can comprehend expository writing and identify spelling, 
punctuation, and grammatical errors; can comprehend a variety of 
materials such as periodicals and nontechnical journals on 
common topics; can comprehend library reference materials and 
compose multiparagraph essays; can listen to oral instructions and 
write an accurate synthesis of them; and can identify the main idea 
in reading selections and use a variety of context issues to 
determine meaning. Writing is organized and cohesive with few 
mechanical errors; can write using a complex sentence structure; 
and can write personal notes and letters that accurately reflect 
thoughts. 


Individual can comprehend, explain, and analyze information from 
a variety of literacy works, including primary source materials and 
professional journals, and can use context cues and higher order 
processes to interpret meaning of written material. Writing is 
cohesive with clearly expressed ideas supported by relevant detail, 
and individual can use varied and complex sentence structures 
with few mechanical errors. 


Individual can perform all basic math functions 
with whole numbers, decimals, and fractions; 
can interpret and solve simple algebraic 
equations, tables, and graphs and can 
develop own tables and graphs; and can use 
math in business transactions. 


Individual can make mathematical estimates 
of time and space and can apply principles of 
geometry to measure angles, lines, and 
surfaces and can also apply trigonometric 
functions. 


Individual is able or can learn to follow simple multistep directions 
and read common legal forms and manuals; can integrate 
information from texts, charts, and graphs; can create and use 
tables and graphs; can complete forms and applications and 
complete resumes; can perform jobs that require interpreting 
information from various sources and writing or explaining tasks to 
other workers; is proficient using computers and can use most 
common computer applications; can understand the impact of 
using different technologies; and can interpret the appropriate use 
of new software and technology. 


Individual is able to read technical information and complex 
manuals; can comprehend some college level books and 
apprenticeship manuals; can function in most job situations 
involving higher order thinking; can read text and explain a 
procedure about a complex and unfamiliar work procedure, such 
as operating a complex piece of machinery; can evaluate new 
work situations and processes; and can work productively and 
collaboratively in groups and serve as facilitator and reporter of 
group work. The individual is able to use common software and 
learn new software applications; can define the purpose of new 
technology and software and select appropriate technology; can 
adapt use of software or technology to new situations; and can 
instruct others, in written or oral form, on software and technology 
use. 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 
CASAS = Comprehensive Adult Student Assessment System e TABE = Test of Adult Basic Education e MAPT: Massachusetts Adult Proficiency Tests ¢ Wonderlic GAIN = Wonderlic General Assessment of Instructional Needs 
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Exhibit 2.1 (Continued) 
Functioning Level Table 


Beginning ESL Literacy 


Test Benchmark: 
CASAS scale scores: 
Reading: 180 and below 
L&W Listening: 162-180 and below 


BEST Plus: 400 and below (SPL 0-1) 
BEST Literacy: 0-20 (SPL 0-2) 


TABE CLAS-E scale scores:* 
Total Reading and Writing: 225-394 
Total Listening and Speaking: 230-407 


Low Beginning ESL. 


Test benchmark: 

CASAS scale scores 
Reading: 181-190 
L&W Listening: 181-189 


BEST Plus: 401-417 (SPL 2) 
BEST Literacy: 21-52 (SPL 2-3) 
TABE CLAS-E scale scores:* 


Total Reading and Writing: 395-441 
Total Listening and Speaking: 408-449 


Outcome Measures Definitions 
EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS—ENGLISH AS A SECOND LANGUAGE LEVELS 


Listening and Speaking Basic Reading and Writing Functional and Workplace Skills 


Individual cannot speak or understand English, or understands 
only isolated words or phrases. 


Individual can understand basic greetings, simple phrases and 
commands. Can understand simple questions related to personal 
information, spoken slowly and with repetition. Understands a 
limited number of words related to immediate needs and can 
respond with simple learned phrases to some common questions 
related to routine survival situations. Speaks slowly and with 
difficulty. Demonstrates little or no control over grammar. 


Individual has no or minimal reading or writing 
skills in any language. May have little or no 
comprehension of how print corresponds to 
spoken language and may have difficulty 
using a writing instrument. 


Individual can read numbers and letters and 
some common sight words. May be able to 
sound out simple words. Can read and write 
some familiar words and phrases, but has a 
limited understanding of connected prose in 
English. Can write basic personal information 
(e.g., name, address, telephone number) and 
can complete simple forms that elicit this 
information. 


Individual functions minimally or not at all in English and can 
communicate only through gestures or a few isolated words, such 
as name and other personal information; may recognize only 
common signs or symbols (e.g., stop sign, product logos); can 
handle only very routine entry-level jobs that do not require oral or 
written communication in English. There is no knowledge or use 
of computers or technology. 


Individual functions with difficulty in social situations and in 
situations related to immediate needs. Can provide limited 
personal information on simple forms, and can read very simple 
common forms of print found in the home and environment, such 
as product names. Can handle routine entry level jobs that require 
very simple written or oral English communication and in which job 
tasks can be demonstrated. May have limited knowledge and 
experience with computers. 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 


CASAS = Comprehensive Adult Student Assessment System e BEST= Basic English Skills Test e TABE CLAS-E = Test of Adult Basic Education Complete Language Assessment System—English 


* Refer to the TABE CLAS-E Technical Manual for score ranges for individual reading, writing, listening and speaking tests. Table shows total scores. 
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Exhibit 2.1 (Continued) 
Functioning Level Table 


Literacy Level 
High Beginning ESL 


Test benchmark: 

CASAS scale scores 
Reading: 191-200 
L&W Listening: 190-199 


BEST Plus: 418-438 (SPL 3) 
BEST Literacy: 53-63 (SPL 3-4) 


TABE CLAS-E scale scores:* 
Total Reading and Writing: 442-482 
Total Listening and Speaking: 450-485 


Outcome Measures Definitions 
EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS—ENGLISH AS A SECOND LANGUAGE LEVELS 


Listening and Speaking 
Individual can understand common words, simple phrases, and 
sentences containing familiar vocabulary, spoken slowly with 
some repetition. Individual can respond to simple questions about 
personal everyday activities, and can express immediate needs, 
using simple learned phrases or short sentences. Shows limited 
control of grammar. 


Basic Reading and Writing 
Individual can read most sight words, and 
many other common words. Can read familiar 
phrases and simple sentences but has a 
limited understanding of connected prose and 
may need frequent re-reading. 


Individual can write some simple sentences 
with limited vocabulary. Meaning may be 
unclear. Writing shows very little control of 
basic grammar, capitalization and punctuation 
and has many spelling errors. 


Functional and Workplace Skills 
Individual can function in some situations related to immediate 
needs and in familiar social situations. Can provide basic personal 
information on simple forms and recognizes simple common forms 
of print found in the home, workplace and community. Can handle 
routine entry level jobs requiring basic written or oral English 
communication and in which job tasks can be demonstrated. May 
have limited knowledge or experience using computers. 


Low Intermediate ESL 


Test Benchmark: 

CASAS scale scores: 
Reading: 201-210 
L&W Listening: 200-209 


BEST Plus: 439-472 (SPL 4) 
BEST Literacy: 64-67 (SPL 4-5) 


TABE CLAS-E scale scores:* 
Total Reading and Writing: 483-514 
Total Listening and Speaking: 486-525 


Individual can understand simple learned phrases and limited new 
phrases containing familiar vocabulary spoken slowly with 
frequent repetition; can ask and respond to questions using such 
phrases; can express basic survival needs and participate in 
some routine social conversations, although with some difficulty; 
and has some control of basic grammar. 


Individual can read simple material on familiar 
subjects and comprehend simple and 
compound sentences in single or linked 
paragraphs containing a familiar vocabulary; 
can write simple notes and messages on 
familiar situations but lacks clarity and focus. 
Sentence structure lacks variety but shows 
some control of basic grammar (e.g., present 
and past tense) and consistent use of 
punctuation (e.g., periods, capitalization). 


Individual can interpret simple directions and schedules, signs, 
and maps; can fill out simple forms but needs support on some 
documents that are not simplified; and can handle routine entry 
level jobs that involve some written or oral English communication 
but in which job tasks can be demonstrated. Individual can use 
simple computer programs and can perform a sequence of routine 
tasks given directions using technology (e.g., fax machine, 
computer). 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 


CASAS = Comprehensive Adult Student Assessment System e BEST= Basic English Skills Test e TABE CLAS-E = Test of Adult Basic Education Complete Language Assessment System—English 


* Refer to the TABE CLAS-E Technical Manual for score ranges for individual reading, writing, listening and speaking tests. Table shows total scores. 
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Exhibit 2.1 (Continued) 
Functioning Level Table 


High Intermediate ESL 


Test Benchmark: 
CASAS scale scores: 
Reading: 211-220 
L&W Listening: 210-218 


BEST Plus: 473-506 (SPL 5) 
BEST Literacy: 68-75 (SPL 5-7) 


TABE CLAS-E scale scores:* 
Total Reading and Writing: 515-556 
Total Listening and Speaking: 526-558 


Advanced ESL 


Test Benchmark: 
CASAS scale scores: 
Reading: 221-235 
L&W Listening: 219-227 


BEST Plus: 507-540 (SPL 6) 
BEST Literacy: 76-78 (SPL 7-8) ** 


TABE CLAS-E scale scores:* 
Total Reading and Writing: 557-600 
Total Listening and Speaking: 559-600 


Outcome Measures Definitions 
EDUCATIONAL FUNCTIONING LEVEL DESCRIPTORS—ENGLISH AS A SECOND LANGUAGE LEVELS 


Listening and Speaking Basic Reading and Writing Functional and Workplace Skills 


Individual can understand learned phrases and short new phrases 
containing familiar vocabulary spoken slowly and with some 
repetition; can communicate basic survival needs with some help; 
can participate in conversation in limited social situations and use 
new phrases with hesitation; and relies on description and 
concrete terms. There is inconsistent control of more complex 
grammar. 


Individual can understand and communicate in a variety of 
contexts related to daily life and work. Can understand and 
participate in conversation on a variety of everyday subjects, 
including some unfamiliar vocabulary, but may need repetition or 
rewording. Can clarify own or others’ meaning by rewording. Can 
understand the main points of simple discussions and 
informational communication in familiar contexts. Shows some 
ability to go beyond learned patterns and construct new 
sentences. Shows control of basic grammar but has difficulty 
using more complex structures. Has some basic fluency of 
speech. 


Individual can read text on familiar subjects 
that have a simple and clear underlying 
structure (e.g., clear main idea, chronological 
order); can use context to determine meaning; 
can interpret actions required in specific 
written directions; can write simple 
paragraphs with main idea and supporting 
details on familiar topics (e.g., daily activities, 
personal issues) by recombining learned 
vocabulary and structures; and can self and 
peer edit for spelling and punctuation errors. 


Individual can read moderately complex text 
related to life roles and descriptions and 
narratives from authentic materials on familiar 
subjects. Uses context and word analysis 
skills to understand vocabulary, and uses 
multiple strategies to understand unfamiliar 
texts. Can make inferences, predictions, and 
compare and contrast information in familiar 
texts. Individual can write multi-paragraph 
text (e.g., organizes and develops ideas with 
clear introduction, body, and conclusion), 
using some complex grammar and a variety of 
sentence structures. Makes some grammar 
and spelling errors. Uses a range of 
vocabulary. 


Individual can meet basic survival and social needs, can follow 
some simple oral and written instruction, and has some ability to 
communicate on the telephone on familiar subjects; can write 
messages and notes related to basic needs; can complete basic 
medical forms and job applications; and can handle jobs that 
involve basic oral instructions and written communication in tasks 
that can be clarified orally. Individual can work with or learn basic 
computer software, such as word processing, and can follow 
simple instructions for using technology. 


Individual can function independently to meet most survival needs 
and to use English in routine social and work situations. Can 
communicate on the telephone on familiar subjects. Understands 
radio and television on familiar topics. Can interpret routine charts, 
tables and graphs and can complete forms and handle work 
demands that require non-technical oral and written instructions 
and routine interaction with the public. Individual can use common 
software, learn new basic applications, and select the correct 
basic technology in familiar situations. 


Note: The descriptors are entry-level descriptors and are illustrative of what a typical student functioning at that level should be able to do. They are not a full description of skills for the level. 


CASAS = Comprehensive Adult Student Assessment System e BEST= Basic English Skills Test e TABE CLAS-E = Test of Adult Basic Education Complete Language Assessment System—English 


* Refer to the TABE CLAS-E Technical Manual for score ranges for individual reading, writing, listening and speaking tests. Table shows only total scores 


** Students can be placed into advanced ESL using BEST Literacy but the test does not assess skills beyond this level so students cannot exit Advanced ESL with this test. Retesting of students who enter this level with another 


assessment is recommended. 
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Definition: Learner completes or advances one or more educational functioning levels from 
the starting level measured on entry into the program. 


Applicable Population: All learners. 


Federal Reporting: Total number of learners who complete a level during the program is 
reported, and a rate or percentage of level completion is computed. The number who continue in the 
program after completing a level, the number who fail to complete a level and leave the program, and 
the number who remain in the same level are recorded to obtain a fuller picture of student flow and 
retention. 


Educational Functioning Levels 


The NRS divides educational functioning into six levels for both ABE and ESL. The levels 
for ABE are beginning literacy, beginning basic education, low and high intermediate basic 
education, and low and high adult secondary education. Each ABE level has a description of basic 
reading, writing, numeracy, and functional and workplace skills that can be expected from a person 
functioning at that level. The six ESL levels are beginning literacy, low beginning ESL, high 
beginning ESL, low and high intermediate ESL, and advanced ESL. The ESL levels describe 
speaking and listening skills and basic reading, writing, and functional workplace skills that can be 
expected from a person functioning at that level. The skill descriptors illustrate the types of skills 
students functioning at that level are likely to have. The descriptors do not provide a complete or 
comprehensive delineation of all of the skills at that level but provide examples to guide assessment 
and instruction. Upon DAEL approval, states may also use additional educational levels and skill 
area descriptions, as long as they are compatible with NRS levels and skills. 


At the low and intermediate levels, the basic reading and writing skills are identical for both 
ABE and ESL. At the higher levels (secondary level for ABE, advanced level for ESL), the reading 
and writing skills are designed to be slightly higher for ABE than for ESL, because the adult 
secondary level is designed to be the highest level. The functional and workplace skills for ABE and 
ESL also differ somewhat by having a stronger second language focus for ESL. Speaking and 
listening skills are only described for ESL, and numeracy is only described for ABE to reflect 
common instructional practice. Programs, however, may apply the numeracy descriptors to ESL 
students and the speaking and listening descriptors to ABE students if the students’ needs and the 
program’s instruction warrant this approach. 


e The descriptors are entry-level descriptors and are illustrative of what a typical student 
functioning at that level should be able to do. They are not a full description of skills for 
a particular level. When a student has skills at one or more levels above the placement 
level, he or she has completed that level and can advance to the next level. 


e Students do not need to be assessed in all of the areas described in the level descriptors. 
The local program must decide, in accordance with State guidelines, the skill areas most 
relevant to each student’s needs or the program’s curriculum and assess students in these 
areas. At a minimum, students must be assessed in basic reading, writing or math. 


e If multiple skill areas are assessed and the student has different abilities in different areas, 
the program should place the student according to the lowest functioning level. For 
example, if a student is at the beginning level in reading and the low intermediate level in 
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numeracy, then the student would be placed in the beginning level. The lowest 
functioning level also should be used to determine educational gain in subsequent 
assessments. 


e An adult high school is a credit bearing secondary education program sanctioned by State 
law, code, or regulation that leads to a secondary school diploma or equivalent. States that 
offer adult high school credit programs (including adult high schools) may measure and 
report educational gain through the awarding of credits or Carnegie units. Adults earning 
credits or Carnegie units in high school level courses can complete low ASE by earning 
enough credits to move to 11" or 12" grade status as determined by state rule or policy. 
Students who enter an adult high school program at 11/12thgrade level (as determined by 
State rules pertaining to credits held) are placed in High ASE regardless of NRS test score. 
An entry educational functioning level of High ASE determines inclusion in the tracking 
cohort for obtaining a secondary credential. Adult students can complete high ASE by 
earning enough credits to complete the requirements for high graduation as determined by 
state rule or policy. These students would be reported as completing high ASE and of earning 
a high school diploma. 


State Responsibilities in Assessing and Measuring Educational Gain 


To measure educational gain within the NRS, States are required to have a written 
assessment policy for its local programs. The assessment policy must identify (1) the tests to be used 
to measure educational gain for both ABE/ASE and ESL students, (2) when pre- and posttests are to 
be administered, and (3) how tests scores are to be tied to the NRS educational functioning levels for 
initial placement and for reporting student advancement across levels. The assessments allowed by 
the State must conform to standard psychometric criteria for validity and reliability and must meet 
the standards provided by DAEL (see below). 


For the educational functioning levels to be meaningful, assessments need to be administered 
in a standardized and consistent way by all programs in each State. When these procedures are not 
followed correctly or consistently, the determination of educational functioning level is invalid and 
not comparable across programs or possibly even within programs, making the data validity 
questionable. Program staff must be trained in test administration and scoring to ensure that the 
measures are valid and reliable across programs and students. 


Assessment of Students in Distance Education 


Students in distance education should be posttested after the same amount of instructional 
time as other students, according to the state’s approved NRS assessment policy. States that choose 
to develop proxy contact hours using one of the approved models will use the proxy contact hours to 
measure the posttest time for distance education students. For example, if the state’s assessment 
policy requires posttesting after 80 contact hours, programs must posttest distance education students 
after 80 proxy contact hours, as determined by the state model. 


States that choose not to collect and report proxy contact hours must develop procedures for 
determining the appropriate time for posttesting students in distance education and may use one of 
the proxy contact hour models or another appropriate method, as long as the posttesting time is after 
the same amount of instructional time as other students. The state will describe the methodology it 
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employed for determining posttest time and procedures for posttesting distance education students in 
its state assessment policy. 


Programs must administer all pre- and post- assessments used to measure educational gain of 
distance education students for NRS reporting in person, at a proctored program site within the state 
that meets NRS assessment policy. Assessments not conducted through face-to-face interaction with 
a trained test administrator in a secure setting are not allowed for NRS reporting. 


Standardized Assessment 


To ensure comparability of the meaning of the educational functioning levels across all 
programs in the State, all programs must use standardized assessment procedures that conform to the 
State’s assessment policy when determining students’ educational functioning levels. The 
assessment procedure must include a standardized test or a standardized performance-based 
assessment with a standardized scoring rubric that has been approved by OCTAE within ED for 
measuring educational gain within the NRS framework. OCTAE conducts the review process 
annually using panels of independent experts in assessment, who evaluate assessments according to 
the process outlined in 34 CFR Part 462 (see Federal Register, Vol. 73, No. 9, January, 14, 2008). A 
list of tests determined suitable for use in the NRS is published annually in the Federal Register. The 
following sections summarize the criteria used to evaluate assessments for measuring educational 
gain for the NRS. 


Intended Purpose of the Instrument 


Generally speaking, tests or other assessment instruments are not inherently valid or invalid; 
rather, their validity hinges on how they are used. Assessments that measure educational gain should 
be designed to measure the development of basic English literacy and language skills through pre- 
and posttesting. This is not to say that tests developed and validated for one purpose can never be 
used for different purposes, only that the converse should not be taken for granted either. Moreover, 
it is usually true that the greater the difference between the intended purpose underlying the 
development of a given instrument and that associated with the needs of the NRS, the less likely that 
the instrument will be suitable for the NRS, regardless of its validity with respect to its original 


purpose. 
Procedures Used to Develop/Maintain the Instrument 


Relevant information associated with the development process includes such details as the 
nature of the sample to which the assessment was administered for the pilot or field testing (e.g., how 
many examinees were administered each item? Were any measures taken to ensure the motivation of 
the examinees? From what population were the samples drawn?), and what steps were taken to 
ensure the quality of the items (e.g., how are items screened for fairness and sensitivity? How are 
they screened for psychometric quality?). With respect to the former, it is of particular relevance to 
ascertain the similarity of the samples used to develop the instrument with that of the adult education 
population. The greater the similarity between the samples used in developing the instrument and the 
population of interest to the NRS, the greater the likelihood that the results associated with those 
samples will generalize to that population. 


Other information associated with the processes used to maintain the assessment that States 
should consider include the rate at which new forms are developed, the steps taken to ensure their 
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comparability with existing forms, and the extent to which security is maintained. It is essential that 
multiple forms of each instrument be available, that scores associated with these forms be equivalent 
in meaning, and that the security of the forms be maintained at all times. 


Matching Instrument Content to NRS Educational Functioning Level Descriptors 


Validity is concerned with the accuracy of measurement; in other words, the extent to which 
the instrument measures what it is intended to measure. Content validity of an assessment is the 
extent to which the items/tasks of the instrument cover the domain of interest. For the NRS, the 
domain of interest is comprised of the skills used to describe the educational functioning levels for 
ABE and ESL. To establish the content validity with respect to the requirements of the NRS, there 
must be evidence that the items/tasks of that instrument measure the skills associated with the educational 
functioning levels (and, by the same token, do not measure skills not associated with the levels). 


Typically, content validity is established via the judgments of subject matter experts (SMEs). 
For instance, a panel of such experts might be asked to judge the extent to which the items/tasks of a 
given instrument require the types and levels of skills described for a particular educational 
functioning level. In general, the greater the judged overlap between the content of the instrument 
and the skills associated with a given level descriptor, the greater the content validity of the 
instrument with respect to its use as a measure of educational attainment at that level. It is important 
to point out that the content validity of a given instrument may vary with respect to different 
educational functioning levels; that is, it may provide adequate coverage of the skills associated with 
some levels but less than adequate coverage of the skills associated with other levels. Finally, it 
should be noted that the usefulness of content validity evidence is directly proportional to the quality 
of the judgments provided. Consequently, the test publisher should establish the credentials of the 
SMEs whose judgments were obtained, including their familiarity with adult education and the NRS 
levels, along with information regarding the number of experts used and the degree of agreement 
among them, both by skill and level. 


Matching Scores on the Instrument to NRS Educational Functioning Levels 


The assessment must provide a way to translate scores on the assessment to the NRS 
educational functioning levels and the method used to establish this translation. States also should 
review the adequacy of the procedures used to establish the translations and the degree of uncertainty 
(or error) associated with them. The process used to identify the level of performance on a given 
instrument that is associated with a given level of achievement in some domain is generally referred 
to as standard setting. Although there are many different approaches to standard setting, most rely 
heavily on the judgments of SMEs. It is important for the test publisher to report the credentials of 
the experts making the standard setting judgments and the number of the experts used and their 
degree of agreement. The latter information is directly related to the degree of error associated with 
the final translations and indicates the extent to which the cut scores to the NRS might be expected to 
differ if they had been established by a different (though similar) panel of experts. The greater the 
degree of agreement is among experts, the greater the amount of faith that can be placed in the 
resulting translations. 


Reliability/Classification Consistency 


Reliability refers to the degree of consistency in performance on an assessment; that is, the 
extent to which an examinee would be expected to perform similarly across multiple administrations 
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of the instrument or under different conditions. An important condition that can differ across 
administrations of a particular instrument to be used for the NRS is the form of the instrument 
administered. More specifically, because educational gain is determined as a function of the 
difference between an examinee’s pre- and posttest performance as measured on different forms of 
the instrument, it is essential to review the test publisher’s information regarding the expected 
similarity of performance across forms in the absence of instruction or other external interventions. 
The greater the similarity in performance across forms, the greater the alternate forms reliability of 
the instrument and the stronger the inference that improvements in performance between pre- and 
posttesting is attributable to something other than measurement error associated with differences 
across forms. 


Note that alternate forms reliability information should be provided for both the raw (or 
number correct) scores associated with the assessment being reviewed and the translated NRS 
educational functioning level classifications. It is the consistency with which examinees are classified 
into the educational functioning levels that is the most important consideration for determining the 
appropriateness of the instrument for use in the NRS, because it is movement across the 
classifications that forms the basis for evaluating educational gain. Also, because the consistency of 
performance measurement may vary with respect to educational functioning levels, information 
regarding classification consistency should be reported for each level that the instrument is being 
considered for use in measuring educational gain. Last, it is important for the test publisher to 
provide information regarding the nature of the sample used to estimate the reliability of the 
instrument because the greater the differences between the sample and the target population (e.g., 
ABE students), the less generalizable the reliability estimates will be. 


Construct Validity 


Other types of validity information that are important in determining the appropriateness of a 
given instrument for measuring educational gain for the NRS fall under the global heading of 
construct validity. 


Convergent validity concerns the extent to which the scores on the instrument are related to 
scores on other instruments designed to measure the same or very similar constructs. States should 
review information provided by test publishers regarding the degree of relationship between 
examinee performance on their instrument and performance on one or more other measures currently 
approved for measuring educational gain in the NRS. This information should be provided with 
respect to the raw scores associated with the assessment and with the corresponding NRS educational 
functioning level classifications. Likewise, information should be provided regarding the nature of 
the sample from which the data were collected to determine the extent to which the results are likely 
to generalize to the population of interest. 


Other types of information States should consider to evaluate construct validity of an 
assessment include evidence regarding the extent to which scores on that instrument are free from 
sources of variance not relevant to the skills the assessment measures, such as practice effects or 
cultural-based knowledge, and the extent to which performance on the assessment is related to other 
variables that it should be related to, such as hours of instruction or other important outcome 
measures (e.g., attainment/retention of employment and acquisition of academic credentials). 


The foregoing is not meant to be an exhaustive list of the types of information that might be 
provided by a test publisher in support of the validity of a given instrument, nor is it meant as a list of 
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information that must be provided. Rather, this guidance is intended to suggest to States the kinds of 
information that would be considered relevant in determining whether a particular instrument is 
appropriate, valid, and reliable for measuring educational gain as a result of participation in an adult 
education program. Exhibit 2.2 summarizes the guidance for evaluating assessments. 


Placing Students in Educational Functioning Levels 


To assist in placement decisions, test benchmarks are provided for the levels. Tests included 
for ABE are CASAS, TABE (forms 9-10), Wonderlic GAIN, and MAPT (for beginning basic 
education and above). For ESL, the test benchmarks include CASAS and scores on BEST Literacy, 
BEST Plus, and TABE CLAS-E. SPLs tied to the BEST and BEST Plus also are included. These 
benchmarks are provided as examples of how students functioning at each level would perform on 
the tests. The tests should not be considered equivalent, however, and do not necessarily measure the 
same skills. 


The NRS requires that local programs assess and place all students into an educational 
functioning level at intake. Programs should administer the initial assessment at intake or within a 
short period thereafter and administer follow-up or posttest assessments according to State policy. 
The follow-up assessment should occur after a set instruction time, either in hours (e.g., after 50 
hours of instruction) or months (e.g., the last 2 weeks of November or the last week of instruction), 
and should conform to the test publisher’s guidelines for the amount of time needed for a student to 
show a meaningful gain. 


Use of Different Assessment Forms 


Assessments designed for multiple administrations on the same students, such as for pre- and 
posttesting, have different but equivalent versions or forms. Pre- and posttesting must use different 
forms. In addition, some tests, such as TABE, have different forms for student proficiency levels, 
designated as “easy” and “hard,” for example. When using such a test, programs must follow the test 
publisher’s guidelines in selecting the correct test form for each student. 


Pretest Administration Time 


The initial assessment is the basis for placing students in an entering educational functioning 
level according to NRS or State definitions. It is the baseline on which programs measure student 
learning gains. Programs should administer the initial assessment to students at a uniform time 
shortly after enrollment. This time should be set by State policy and apply to all students to improve 
test comparability among students. If available, programs should administer a locator test for 
guidance on the appropriate pretest to use. 


Placement Policy Based on Initial Assessment 


Using the results of the initial assessment, programs should place students at the appropriate 
NRS educational functioning level or the equivalent State level. States should provide to local 
programs the criteria for placing students at each educational functioning level, using test scores from 
the initial assessment. Not all of the skill areas described in the level descriptors need to be used to 
place students, but the skill areas assessed should be the areas most relevant to the students’ needs 
and the program’s curriculum. If multiple skill areas are assessed and the student has differing 


NRS Implementation Guidelines 27 


Chapter II. NRS Measure Definitions and Data Collection Methods 


abilities in each area, however, NRS policy requires that the program place the student according to 
the lowest skill area (as discussed earlier in this report). 


Established Time for Postassessment 


Just as programs should administer the initial assessment to students at a uniform time, the 
State also should establish a time for posttesting. This time may be after a set number of 
instructional hours or months of instruction and should be long enough after the pretest to allow the 
test to measure gains. As noted earlier, local programs must conduct posttests with the parallel form 
of the same assessment used to place the student. 


Level Advancement Policy Based on Postassessment 


Educational gain is determined by comparing the student’s initial educational functioning 
level with the educational functioning level measured by the posttest. To allow local programs to 
determine gain, the State must use the educational functioning level definitions and correlate 
assessment scores to specific levels. It is important to note that if a student is not posttested, then no 
advancement can be determined for that student. The student must remain in the same level as 
initially placed for NRS reporting. 


Staff Training on Administration of Assessments 

The State should ensure that all local program staff who administers assessments receives 
training on proper administration procedures. Such training should be provided on an ongoing basis 
to accommodate new staff and as a refresher to staff who had earlier training. These procedures 
include the steps outlined above (i.e., use of the correct form of the assessment and administration at 
the proper time) and also include following the publisher’s procedures for giving directions to 
students, timing the assessment, and not providing help to students. Assessments also should be 
administered under good conditions (e.g., in a well-lit, quiet room). 


Exhibit 2.3 summarizes assessment guidelines for measuring educational gain for the NRS. 


FOLLOW-UP OUTCOME MEASURES 


The NRS follow-up measures are outcomes that students may achieve at some time following 
participation in adult education. These measures are: 


e Entered employment—whether the student obtained a job. 
e Retained employment—whether the student remained in the job. 
e Receipt of a secondary school credential or its equivalent. 
e Entered postsecondary education or training. 
States are not required to collect all of the follow-up measures on all students but only on 


students who are members of the cohort to which the measure applies. These measures are defined in 
this section, and the procedures for collecting them are presented. 
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Exhibit 2.2 
Guidance for Evaluating Assessments Used for Measuring Educational Gain 


What is the intended purpose of the instrument? 


a. What does the instrument’s technical manual say about the purpose of the instrument, and how does this match the requirements of the NRS? (The NRS needs instruments that 
allow examinees to demonstrate their standing on skills represented in the educational functioning level descriptors. It also needs instruments for which multiple parallel forms exist, 
so that gains in educational functioning can be demonstrated.) 


What procedures were used to develop and maintain the instrument? 


b. How was the instrument developed? (How similar was the sample[s] of examinees used to develop/evaluate the instrument to the population of interest to the NRS? What steps, if 
any, were taken to ensure their motivation while responding to the instrument? To what extent have items/tasks on the instrument been reviewed for fairness and sensitivity? To 
what extent have they been screened for adequacy of psychometric properties? Does the instrument have multiple forms?) 

How is the instrument maintained? (How frequently, if ever, are new forms of the instrument developed? What steps are taken to ensure the comparability of scores across forms? 
What steps are taken to maintain the security of the instrument?) 


Does the assessment match the content of the NRS educational functioning level descriptors? 


d. How adequate are the items/tasks on the instrument at covering the skills used to describe the NRS’ educational functioning levels? Are aspects of a given descriptor not covered 
by any of the items/tasks? Are there items/tasks not associated with any of the descriptors? (Note: it is possible for an instrument to be appropriate for measuring proficiency at 
some levels but not at others.) 

What procedures were used to establish the content validity of the instrument? How many SMEs provided judgments linking the items/tasks to the educational functioning level 


descriptors, and what were their qualifications? To what extent did their judgments agree? 
Can the scores on the assessment match the NRS educational functioning levels? 


f. What standard setting procedures were used to establish cut scores for transforming raw scores on the instrument to estimates of an examinee’s NRS educational functioning 
level? If judgment-based procedures were used, how many SMEs provided judgments, and what were their qualifications? To what extent did their judgments agree? 


g. Whatis the standard error of each cut score, and how was it established? 


Is there evidence of reliability and classification consistency? 


h. What is the correlation between raw scores across alternate forms of the instrument? What is the consistency with which examinees are classified into the same NRS educational 
functioning level across forms? 

i. | How adequate was the research design that led to these estimates? (What was the size of the sample? How similar was the sample used in the data collection to that of the adult 
education population? What steps were taken to ensure the motivation of the examinees?) 


Has construct validity of the assessment been demonstrated? 


j. To what extent do scores (and/or educational functioning classifications) associated with the instrument correlate (or agree) with scores or classifications associated with other 
instruments already approved by ED for assessing educational gain? To what extent are they related to other relevant variables, such as hours of instruction or other important 
process or outcome variables? How adequate were the research designs associated with these sources of evidence? 


What other evidence is available to demonstrate that the instrument measures gains in educational functioning resulting from adult education and not some other construct- 
irrelevant variables, such as practice effects? 
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Exhibit 2.3 
Summary of Assessment Guidelines for Measuring Educational Gain 


7 
e i : 
| 


Develop a level advancement policy based on the posttest or follow-up assessment 
Train staff in administrating the assessments. 


Establish a uniform time for posttest based on test publisher’s guidelines. 


Follow-up Measure #1: Entered Employment 


Definition: Learner enters employment by the end of the first quarter after the program exit 
quarter. Employment is working in a paid, unsubsidized job or working 15 hours or more per week 
in an unpaid job on a farm or business operated by a family member or the student. The exit quarter 
is the quarter when instruction ends, the learner terminates or has not received instruction for 90 
days, and is not scheduled to receive further instruction. A job obtained while the student is enrolled 
can be counted for entered employment and is reported if the student is still employed in the first 
quarter after exit from the program. 


Applicable Population: Learners who are not employed at time of entry and in the 
labor force who exit during the program year. 


Federal Reporting: States report the total number of unemployed learners in the labor 
force who enter employment and exit during the program year and the total number of learners in 
the labor force who are unemployed at entry who exit during the program year. Entered 
employment rate is computed by dividing these numbers. 


Follow-up Measure #2: Retained Employment 
Definition: Learner remains employed in the third quarter after exit quarter. 


Applicable Population: Learners who, at time of entry, were not employed and in the 
labor force, who are employed in the first quarter after exit quarter, and learners employed at 
entry. 


Federal Reporting: States report the total number of learners in the applicable 
population (see above) who are employed in the third quarter after program exit and the total 
applicable population (i.e., the number of learners in the labor force who were unemployed at 
entry, and who entered employment; and learners who are employed at entry). The retained 
employment rate is computed by dividing the number of learners who retain employment by the 
total applicable population. 


NRS Implementation Guidelines 30 


Chapter II. NRS Measure Definitions and Data Collection Methods 


Follow-up Measure #3: Receipt of a Secondary School Diploma 


Definition: The learner obtains certification of attaining passing scores on a State 
recognized high school equivalency test, or the learner obtains a diploma or State-recognized 
equivalent, documenting satisfactory completion of secondary studies (high school or adult high 
school diploma). 


Applicable Population: All enrolled learners who take all parts of a State recognized 
high school equivalency test, or are enrolled in adult high school” at the high ASE level, or are 
enrolled in the assessment phase of the EDP who exit during the program year. 


Federal Reporting: States report the total number of learners in the applicable 
population (see above) and the number of exited learners who pass all parts of a State 
recognized high school equivalency test or obtain secondary school diplomas. To compute a 
rate or percentage of attainment, the number of students receiving a secondary school diploma or 
pass a State recognized high school equivalency test is divided by the total number of learners in 
the applicable population who exit during the program year. If a State has a policy officially 
recognizing passing a foreign language high school equivalency test as receipt of a secondary 
school diploma or its recognized equivalent, the State may also report passing a foreign language 
high school equivalency test in the NRS for adult education. A passing score on a State 
recognized high school equivalency test is defined by the State and must be consistent with State 
policies relating to the issuance of a high school diploma or its equivalent. 


Follow-up Measure #4: Entered Postsecondary Education or Training 


Definition: Learner enrolls in a postsecondary educational, occupational skills training 
program, or an apprenticeship training program that does not duplicate other services or training 
received, regardless of whether the prior services or training were completed. 


Applicable Population: All learners who passed a State recognized high school 
equivalency test or earned a secondary credential while enrolled in adult education, or have a 
secondary credential at entry, or are enrolled in a class specifically designed for transitioning 
to postsecondary education who exit during the program year. A transition class is a class that 
has a specific purpose to prepare students for entry into postsecondary education, training or an 
apprenticeship program. 


Federal Reporting: The total number of learners who enter postsecondary education or 
a training program and the total number of students in the applicable population (see above) 
who exit during the program year are reported. Entry into postsecondary education or training 
can occur any time from the time of exit till the end of the following program year. To 
compute a rate of placement, the number of students enrolling in postsecondary education or 
training, through the end of the program year following the year of program exit, is divided by 
the total number of learners in the applicable population who exited during the program year. 


* See the definition of adult high school on page 25. 
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GUIDANCE FOR COLLECTING THE FOLLOW-UP MEASURES: 
SURVEY METHOD 


The NRS offers two methods for collecting the follow-up measures: a local program follow- 
up survey and data matching. The follow-up survey can be conducted on all or a representative 
sample of learners for whom the outcome measures apply. 


For the enter employment measure, local programs, the State, or a third-party contractor 
conducts data match or a survey in the first quarter after the exit quarter. Retained employment data 
is collected in the third quarter following exit. The other follow-up measures may be collected at any 
time up to the reporting deadline (December 31). For the measure of entry into postsecondary 
education or training, states can report students who enter postsecondary education training through 
the end of the program year following the year of program exit. 


Within the NRS, States may use either method, or a combination of both, to collect follow-up 
measures. For example, the measure of secondary credential attainment could be collected by data 
matching and the remaining measures could be collected by survey. The following section describes 
the general requirements and procedures for employing the two methods. Appendix A also contains 
model surveys and guidance on how to conduct a survey. 


Conducting a Local Follow-up Survey 


Section 231(e) (2) of WIA requires that States assess local program performance on the core 
WIA indicators, which are the four NRS core follow-up measures. Consequently, States must obtain 
these measures on appropriate student cohorts in each of their adult education programs. However, 
States that use the sampling methodology described in this section may make state-level estimates. 
States electing to collect follow-up measures through a survey (as opposed to data matching) should 
follow the procedures summarized below. 


Universe or Sample Survey 


It is advisable to include all students in a follow-up survey; that is, programs should include 
the universe of learners. For programs serving large numbers of students with follow-up outcome 
measures, however, the NRS does allow States the option to include a representative sample of the 
program learners in the survey. 


Sampling a representative group of students can be much less expensive than a survey of all 
students but it creates a degree of error in the findings from the survey, known as sampling error. 
Sampling error becomes greater the lower response rate, making it-difficult to estimate the true value 
of the outcome measure for all students. Since the response rate for adult education students can be 
low, including the universe of learners in the survey rather than a sample could be advantageous 
because there will be no sampling error. Since most States and adult education programs have 
minimal resources to conduct a full survey, the number of students required for accurate results may 
make sampling an attractive option for large programs. 


With these considerations in mind, the NRS requires States choosing to collect follow-up outcome 
measures from a representative sample of participants to adhere to the following procedures. 
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e Estimate the State Cohort Size. Use NRS data from the most recent years available to 
estimate cohort sizes for the State. It is recommended that you use the previous two years’ 
data to get a more stable estimate, if possible. States should create the cohorts according to 
the NRS definitions. 


e Set the Minimum State Sample Size (n ,) for Each Outcome Cohort. Use the NRS sample 
size table provided on the NRS web site to determine the State sample size. The sampling 
tables list the n, required for each category of possible cohort sizes. The sample sizes in the 
table are designed to produce samples with enough power to detect a 5 percent difference or 
change (e.g., a change in the percent of students gaining employment from one year to the 
next) with a 70 percent response rate. 


e Estimate the Cohort Size for Each Follow-up Measure for Each Program. Use NRS data 
from each local program from the most recent years available to estimate cohort sizes within 
each program. As with State cohort sizes, it is recommended that you use the previous two 
years’ data to get a more stable estimate, where possible. Create the cohorts according to the 
NRS definitions. 


e Set the Sample Size for Each Program in the State. Within the state, set the minimum 
sample size required for each local program in the state proportionate to the size of the cohort 
in that program in comparison to the State total. For example, if the program has 10 percent 
of all students in the cohort, the sample size for that program would be 10 percent of the total 
sample. In some cases, the program sample size could vary slightly or be equal to the total 
number of students in the cohort in that program. In such cases, the program should include 
all students in the survey. 


e Randomly Sample the Required Sample Size for Each Program. Provide each program 
with the student sample to enable local staff to conduct the survey. States may also conduct 
the survey at the state level. The required response rate for a survey conducted with a 
random sample is 70 percent. 


For example, if a state has 5,000 students in the cohort, the sample size would be 910 (see table 
provided on the NRS web site). The state would then draw a random sample from the cohort 
from each local program in the state, proportional to the relative size of the cohort to the state 
total. For example, if a program contributes 1,000 students to the cohort, or 20 percent of the 
state total (i.e., 1000/5000), that program’s sample size would be 182 (i.e., 20 percent of the total 
state sample size of 910). 


The procedures above will allow for a statewide estimate of performance for the follow-up measures. 
States may also sample to obtain individual program estimates of performance. However, these 
additional estimates are optional and States interested in making such estimates should contact 
OCTAE for further guidance. 
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Time Period for Conducting the Survey 


The survey may be conducted by the State, local programs, or a third-party contractor, as long as a 
program-specific sample is used. The entered employment measure must be collected from students 
who leave the program by the end of the first quarter after they exit. A job obtained while the student 
is enrolled can be counted for the entered employment measure, but it is still measured and reported 
in the first quarter after the student exits. Retained employment must be collected in the third quarter 
following exit from the program, for students who entered employment according to the NRS 
definition and employed students who exit. There are no time periods tied to the other follow-up 
measures, thus they may be collected at any time until the end of the reporting period (December 31). 
States may include students who enter postsecondary education training up to the end of the next 
Program Year (June 30). 


Exhibit 2.4 summarizes the times at which data are to be collected and the student population 
to which each core follow-up outcome measure applies. 


Exhibit 2.4 
Student Population and Collection Time for Core Follow-up Measures 


Core Outcome Measure Student Population to Include Time Period to Collect Measures 
Learners unemployed at entry andinthe | -. F 
Entered employment labor foros whoreki First quarter after exit quarter 


Learners unemployed at entry in the 
labor force who exit and are employed 
during first quarter after exit; and 
learners employed at entry who exit 


Retained employment Third quarter after exit quarter 


Learners who earned a secondary 

credential while enrolled, have a Any time after exit through the end of 
Placement in postsecondary education | secondary credential at entry or who are | the program year (June 30) following 
or training enrolled in a class specifically designed | the year of program exit. 

for transition to postsecondary education 

who exit. 


Learners who take all parts of a State 
recognized high school equivalency test, 
are enrolled in adult high school at the 
high ASE level or are enrolled in the 
assessment phase of the EDP who exit. 


Any time after exit to the end of the 
reporting period (December 31) 


Receipt of a secondary school 
credential or its equivalent 


* For all measures, exit quarter is the quarter when the learner completes instruction or has not received instruction 
for 90 days and has no instruction scheduled. A job obtained while the student is enrolled can be counted but must 
be reported and measured during the first quarter after exiting the program if the student remains employed in that 

quarter. 


Since the entered and retained employment measures are tied to calendar quarters, the 
simplest approach may be to conduct the survey quarterly. If quarterly collection is conducted, the 
survey should begin no sooner than the last month of the quarter and be completed within 3 months 
(one quarter). Attaining measures for secondary credential and entering postsecondary education is 
not time bound. Although such data can be collected at any time during the reporting period, the 
easiest option is to collect it by quarter. The program or State should determine the optimal time to 
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collect these measures. For example, it may be advisable to collect the entry into postsecondary 
measure in the fall quarter when most students enter postsecondary education. If there are scheduled 
times when high school equivalency tests are given, then the program could collect the secondary 
credential measure to concur with these times. It is recommended that equal numbers of students be 
surveyed each quarter to allow for respondent variations that may occur over the year. For example, 
if 300 students are to be surveyed, about 75 students should be surveyed each quarter. 


Quarterly data collection is strongly recommended (see Exhibit 2.5), especially for the 
employment measures, but States may survey more frequently, if the time period is more convenient 
or cost efficient. For example, the program could conduct continuous, ongoing, or monthly surveys. 
For the other follow-up measures, quarterly data collection is recommended but could also be 
conducted at the end of semesters or instructional periods (such as in December and June) to 
correspond more closely to high school equivalency testing dates or postsecondary education 
enrollment times. The time lag to contact students after they exit the program, however, should be as 
short as possible: The longer the time, the greater likelihood of a lower response rate (since some 
students will move) and the resulting effect of less valid data. 


Most programs and States consider conducting the follow-up survey the most difficult aspect 
of NRS data collection. It is difficult to conduct a survey in a way that produces valid and reliable 
results. The process includes determining which students you must include in the survey; sampling 
students, if necessary; locating them and securing their cooperation; and administering the survey. 
Finding the students and getting them to cooperate in the survey is critical to its success since the 
response rate—the proportion of students you reach—largely determines the validity of the 
information. Locating adult education students is especially difficult, given the transient nature of 
many adult education students. The procedures described below will assist States in conducting a 
valid survey. 


Exhibit 2.5 
Quarterly Periods for Collecting Entered and Retained Employment 


, Collect Entered Employment by Collect Retained Employment by 
Ex Quartet the End of: the End of: 
First Quarter 
(July 1-September 30) Second Quarter Fourth Quarter 


Second Quarter : First Quarter, 
October 1-December 31 Third Quarter Next Program Year 
i) 
Third Quarter Second Quarter, 
(January 1-March 31) pour ane Next Program Year 
Fourth Quarter First Quarter, Third Quarter, 
(April 1-June 30) Next Program Year Next Program Year 


Method for Identifying Follow-up Students 


The local program’s database must have the ability to identify students who should be 
followed, including (1) all students in the cohort applicable to each measure (see definitions) who 
exit, (2) student identification number and contact information, (3) the follow-up outcome that 
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applies to the student and (4) the date that the student left the program. This information needs to be 
retrievable quarterly or according to the time when surveys are to be administered. 


State Survey Instrument 


In any survey, how the questions are asked may influence the responses. Therefore, it is 
important that the survey questions asked do not bias or affect responses. For comparability of data 
among programs in the State, it is also highly advisable that all programs in the State use the same or 
equivalent survey instruments. The State should provide all programs with a standard survey 
questionnaire that is short and simple. It is not necessary to have a long or complicated survey to 
collect NRS measures. For example, it is only necessary to ask if the person got a job or passed a 
high school equivalency test. In addition, the survey should be translated into the most common 
languages spoken by students in local programs. Appendix A provides model surveys designed to 
collect NRS follow-up measures. The models are offered to guide States in designing and 
conducting the follow-up survey and are not required. 


Local Resources to Conduct Surveys 


Conducting a survey is labor intensive. Besides administering the survey, students must be 
located, the survey needs to be explained to them, and their cooperation must be obtained. This work 
requires frequent callbacks to students and careful recordkeeping. States should ensure local 
programs have sufficient staff and time to conduct the survey. Another approach is to have the survey 
conducted for all programs centrally at the State level, either by State staff or by contract to a third 
party. Although costly, this approach is desirable because it removes much of the burden from local 
programs. 


Staff Trained on Surveying 


Like any other data collection effort, staff must follow a uniform set of procedures to collect 
data in a valid and reliable manner. Staff conducting the survey must be trained in its administration, 
including what to say to students to introduce the survey and get their cooperation, ways to avoid 
refusals, how to ask the survey questions, how to record responses, and how to answer student 
questions about the survey. Staff should be thoroughly familiar with all questions and procedures 
before beginning. 


Procedures to Improve Response Rate 


The validity of a survey depends largely on the response rate—the proportion of people who 
respond to the survey out of the total number targeted for the survey. The NRS requires a minimum 
response rate of 50 percent for a universe survey and 70 percent for a representative sample survey. 
Getting a good response rate is probably the most difficult part of conducting a survey, and it may be 
especially hard for adult education students because many are transient and may not have telephones 
or are otherwise difficult to locate. 


To help improve response rate, it is very important that students know they may be contacted 
later and asked about their outcomes. Programs should inform students at program entry about the 
survey and collect extensive contact information about them, such as addresses and phone numbers 
of relatives or others who may know the students’ whereabouts over time. In addition, students 
should be encouraged to provide new addresses and phone numbers when they move, and programs 
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should implement procedures to update this information periodically while the student remains 
enrolled. These procedures can greatly assist in locating students months later when the survey is 
conducted. States should provide local programs with additional guidance to improve response rates. 


Database and Procedures for Survey Reporting 


The State or local programs need a database to keep track of which students are to be 
contacted for the survey, which students have been reached, and whether the students achieved the 
outcomes. This information is needed to conduct the survey and track response rates. The State 
needs the information so it can aggregate the data among programs for NRS reporting. The State 
must report to ED the overall State percentage of students who achieved each of the follow-up 
outcomes. 


To compute the State overall measures for each outcome, the State has to aggregate each of 
the measures from every local program to compute an average. Each local program must report the 
following information to the State to enable computation of the State average: 

e Total number of students in each outcome group that exited during the year. 


e The total number of students sampled, if the program used sampling. 


e Number of students who responded to the survey (the realized or actual sample size) and the 
response rate. 


e The number of students who achieved each outcome. 


Exhibit 2.6 summarizes the guidelines for conducting the follow-up survey. 


Exhibit 2.6 
Summary of Follow-up Survey Guidelines 


1. Develop a method for identifying students to contact for follow-up. 
2. Establish State sampling procedures, if appropriate. 
. Conduct the survey at a proper time. 


3 
4. Ensure that the State has a uniform survey instrument. 
5 
6 


. Train staff to conduct the survey. 


. _ Identify local resources available to conduct the survey. 
7. Implement procedures to improve response rates. 
8. Ensure that the State has a database and procedures for survey reporting. 
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GUIDANCE FOR COLLECTING FOLLOW-UP MEASURES: 
DATA MATCHING 


A second method States can use to collect NRS follow-up measures is data matching. Data 
matching refers to the procedure where two or more State agencies pool and share data on a 
common group of participants. The data consist of individual student records collected by each of 
the agencies that can be linked through a common identifier, typically a Social Security number. 
Matching the pooled data using the common identifier produces a new individual student record or 
an aggregated data report containing data from one or more of the additional agencies. Each agency 
can use the new, pooled data records or reports to understand the impact of its program on 
participants and to obtain data to meet its reporting and accountability requirements. 


Data matching methods are particularly well suited for studying outcomes that occur some 
time after program participation. For example, wage record information systems are used to study 
the outcomes of vocational education and employment programs. The WIA requires job training 
programs funded under Title I to use a data matching methodology to obtain the required 
employment outcomes. Although not required by WIA for Title II programs (adult education), the 
data matching methodology is an efficient way to collect the core follow-up measures. 


Several reasons make data matching attractive. The first major advantage of data matching is 
that it is significantly less costly than the local survey methodology. The costs of conducting a 
survey—drawing a sample, training interviewers, making phone calls—are replaced with the much- 
reduced cost of combining, cleaning, and analyzing the data. Further, this cost can be divided among 
the participating agencies. 


The second major advantage of data matching is reduced data collection burden. At the local 
program level, staff no longer needs to conduct survey procedures. Local programs collect only the 
demographic, participation, and educational functioning level information. Matching is then done at 
the State level. 


Finally, matched data are likely to be more valid than those collected through surveys, which 
are self-reported data. For example, the wage or unemployment record database would reveal 
whether students have actually worked. In addition, response rates for surveys are typically low, 
limiting the amount of information available on a substantial percentage of students. With data 
matching, considerably fewer students are missed, provided each agency has valid Social Security 
numbers. However, the need for Social Security numbers makes data matching problematic in some 
States because of confidentiality issues. Some States have laws against interagency sharing of Social 
Security numbers, and some students are reluctant to give such information to Government agencies. 


Data Matching Models 


Under a data matching system, each participating agency collects a common core of 
demographic and descriptive information on their participants, dates of program participation, a 
common identification number (Social Security number), and the outcome measures specific to its 
program. All measures that are shared among the agencies need to have common definitions for the 
resulting analyses and reports to be meaningful for agencies. 
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There are two data matching models. Under the central data processing or data warehousing 
model, each agency submits to a central source (either a contractor or in-house agency) its individual 
client records containing the data to be shared. This central agency combines the information into a 
single data pool and eliminates record duplication using Social Security numbers. This data pool is 
then available to the individual agencies, which can request specific tables and reports. The reports 
are usually in aggregate form at the State, program, and site levels, although individual data reporting 
can be produced. Local program providers can also request reports through their agencies. Exhibit 
2.7 shows the data warehouse model. Under a second, decentralized or data harvesting model of data 
matching, each agency maintains its own data records and each separate agency requests matches 
from the agency with the needed data. To match with an outside agency, the requesting agency sends 
to the other agency the records containing Social Security numbers and other data needed for the 
analysis, along with the format of the data tables needed. The other agency makes the matches and 
reports the data in the requested format. 


For example, to obtain the high school equivalency test results of students, the State could 
send to the State agency that does high school equivalency testing the program information and 
Social Security numbers and demographic information of students who have taken the high school 
equivalency test. The testing agency matches the records to produce a report on the number and 
characteristics of students who have passed the high school equivalency test. The State could then 
use this information in its annual NRS reporting. 


For both types of data matching, incorrect or missing Social Security numbers affect the 
availability of data. This problem can be substantial if students refuse to provide their Social 
Security numbers or provide incorrect numbers. Legal barriers to collecting Social Security numbers 
also pose a significant barrier to this methodology. Another serious problem affecting data analyses 
with data matching is the time lag from the end of the reporting period to the point at which the data 
are available. It often takes two or more quarters for all of the data to be available. In States using 
data matching, the time lag ranges from one quarter to a year. For example, if a student leaves the 
program in February, the entered employment outcome would need to be measured in the next 
quarter (April-June). If the time lag is two quarters, however, that student’s entered employment 
cannot be determined until the first quarter of the following calendar year. 
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Exhibit 2.7 


Example of Shared Interagency Database—Data Warehouse 


Adult Education Program 


© Educational Level 
© Demographics 
@ Contact Hours 


Employment Program 


e@ Enrollment 


@ Type of Training 
@ Service Hours 
@ Demoaranhics 


Welfare Program 


© Welfare Benefits Received 
© Demographics 


*HSE = High School Equivalency 


HSE* or Secondary Wage Records 


Testing Agency 


Quarterly Wages 
@ HSE* Test Results 
@ Diploma 


Combined 
Participant 
Record or Report 


Shared 
Interagency 
Database 


Postsecondary Institutions 


@ Enrollment 
@ Demographics 
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Implementing Data Matching 


Several States currently use data matching procedures to collect outcomes on adult education 
students. This section provides a discussion of some of the major issues that need to be addressed to 
develop data matching to collect NRS follow-up measures. 


Data matching arrangements are difficult to establish and require considerable time to 
implement. For States using data matching, it took from 2 to more than 5 years to implement the 
process. Crucial to the implementation of the methodology is an interagency planning process with 
individuals committed to system development. This process is successful when political concerns 
are kept out of planning and development. Another essential ingredient is for each agency to have an 
automated, individual participant record system. It is not necessary, however, that each agency use 
the same record system or software, only that the software used by each agency produces information 
in a common format to allow data matching. 


Beyond these basic planning and infrastructure needs, there are three conceptual problems 
that need to be surmounted to develop shared data arrangements: 


e Common outcome and measure definitions. 
e Concerns about data confidentiality. 
e Training and technical assistance. 


The management information system (MIS) must have common definitions for measures that 
are shared. Agencies with jurisdiction over different types of programs (e.g., Departments of Labor 
and Education) must provide data that are based on common understandings of the measures. 
Furthermore, agencies within a single department (e.g., community colleges and local education 
agencies) must also use common definitions. Care must be taken, however, to ensure that the 
definitions agreed upon maintain their fidelity with the mission of the program. If common 
agreement on common definitions cannot be reached, each agency must understand what the other 
definitions are and must be able to accommodate these differences in interpreting the data. For 
example, if program completion is a common data element, each agency must use the same definition 
or must have an understanding of what the other definitions are and interpret the data accordingly. 


States using data matching must comply with the Code of Federal Regulations (CFR) Title 
34, Part 99 — Family Educational Rights and Privacy. The regulations in 34 CFR §99.31, which were 
published on December 2, 2011, articulate the specific conditions under which information may be 
disclosed or shared. Some States may have laws against not only sharing Social Security numbers or 
other unique identifiers but also against sharing educational records. These issues must be addressed 
before data matching can become a widely used methodology for the NRS. 


Finally, a great deal of training and technical assistance at the local level is needed to develop 
a system that produces valid and reliable data. Training needs to be provided on measure definitions, 
data collection and reporting, and data use. Such training also produces buy-in to the whole data 
collection and analysis process and can help “convert” teachers, local staff, and other stakeholders 
who might be skeptical about the usefulness of the system. The training also can supply local 
providers with an idea of how the data are used at the State level, and how they can use it to improve 
their programs. 
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Technical Guidance for Data Matching 


Data matching is a technical process that requires the data system to produce specific data in 
a required format. To conduct this process, the State or local programs must have a database able to 
perform the functions described in this section. 


Procedures to Collect and Validate Unique Identifiers 


Data matching works by pairing records from different databases for the same student using a 
common identifier—a Social Security number or some other unique identifier. Consequently, a valid 
Social Security number or unique identifier must be obtained for all students whose data is in the data 
matching pool. This number is usually collected at intake, and in some States and localities, students 
need to be informed about the use of their numbers for this purpose. Some States may require 
written permission. It is critical to obtain Social Security numbers or unique identifiers, because 
without them, data cannot be matched and outcomes cannot be reported. Similarly, there must be a 
process to verify the validity of Social Security numbers or unique identifiers for matching. The 
State or local program database must be able to produce a report to identify students with missing, 
erroneous, or duplicate Social Security numbers or unique identifiers. 


Common Format for Matching 


There are several ways to perform data matching, and all techniques rely on software to link 
multiple databases and produce the number of matches for each outcome area. To perform these 
operations, the software requires State and local data to be in a specific format that includes the 
location, size, and name of each variable, as well as the technical format in which the local program 
database is to write the data. States must ensure that program databases can produce the data 
according to States’ specifications and that programs submit data in this format or in a way that it can 
be converted to this format. 


Time Period for Data Matching 


The State should have a standard time period for data submission, such as quarterly or 
annually. Data submitted for matching should include the exit data and be for the correct exit 
quarters according to NRS definitions. There also should be checks to ensure that local data do not 
include students who are still enrolled or students who exited in other time periods. 


Data System Produces Individual Students Records 


Successful data matching requires individual student records with three pieces of 
information: (1) a Social Security number or unique identifier, so that data can be linked across 
databases; (2) the outcome measure applicable to the student (e.g., enter employment) or separate 
files for students with each outcome on which data will be matched, so that the student can be 
matched with the correct database; and (3) the exit quarter for employment outcomes because the 
NRS requires entered employment to be measured in the first quarter after the exit quarter. Retained 
employment must be measured during the third quarter after exit quarter. The database must be 
capable of producing records with at least this information and in the State’s required format, as 
discussed previously. 
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CORE DEMOGRAPHIC, STATUS, AND PARTICIPATION MEASURES 


The NRS includes required descriptive measures, which are student demographics and 
student status in several areas. These measures allow for a description and understanding of who 
attends adult education programs and for what reasons. The measures also facilitate analyzing the 
performance of students attending adult education, such as unemployed students or students 
receiving public assistance. The demographic measures include ethnicity, age, gender and highest 
education level or credential attained. Also, there are two descriptive measures for teachers: years of 
experience and teacher certification. The status measures include employment status and whether the 
student has a disability or is on public assistance. 


There are two participation measures—contact hours and program enrollment type— 
collected for both descriptive and analytic purposes. These measures record the amount of 
instruction that students receive and the number of students who attend in areas such as family 
literacy and workplace literacy. This section provides definitions of these measures and guidelines 
for collecting them. 


Demographic and Status Measure Definitions 


Adult education programs always collect NRS demographic and status measures. Program 
staff either collects these measures from the student at intake into the program, or the student directly 
reports these measures, as defined below. 


Demographic Measure #1: Race/Ethnicity 


In 1997, the U.S. Office of Management and Budget (OMB) published Revisions to the 
Standards for the Classification of Federal Data on Race and Ethnicity in the Federal Register, 
Volume 62, Page 58782 (October 30, 1997). The new categories separate race and ethnicity and 
include two categories for data on ethnicity. ED released Final Guidance on Maintaining, 
Collecting, and Reporting Racial and Ethnic Data to the U.S. Department of Education in the 
Federal Register, Volume 72, Page 59266 (October 19, 2007). The following definitions and 
procedures adhere to these requirements. 


Definition: Racial or ethnic category to which the learner self-identifies, appears to belong 
to, or is regarded in the community as belonging. 


When collecting data, program staff are to first ask about a student’s ethnicity (i.e., 
Hispanic/Latino or not) and then select one or more races with which the student identifies. 
Programs report data by counting students in only one of the following seven aggregate racial/ethnic 
categories: 


e American Indian or Alaska Native—A person having origins in any of the original 
peoples of North and South America (including Central America), and who maintains a 
tribal affiliation or community attachment. 


e Asian—A person having origins in any of the original peoples of the Far East, Southeast 
Asia, or the Indian subcontinent including, for example, Cambodia, China, India, Japan, 
Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam. 
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e Black or African American—A person having origins in any of the Black racial groups 
of Africa. 


e Hispanic/Latino of any race—A person of Cuban, Mexican, Puerto Rican, South or 
Central American, or other Spanish culture or origin, regardless of race. The term 
“Spanish origin" can be used in addition to “Hispanic/Latino or Latino." 


e Native Hawaiian or Other Pacific Islander—A person having origins in any of the 
original peoples of Hawaii, Guam, Samoa, or other Pacific Islands. 


e White—A person having origins in any of the original peoples of Europe, the Middle 
East, or North Africa. 


e Two or more races—A person having origins in two or more race categories and not 
Hispanic/Latino. 


Students who identify themselves as Hispanic/Latino are reported only in that category. 


For more information on how to implement these new race/ethnicity data collection and 
reporting requirements, please refer to the following resources. 


e "Revisions to the Standards for the Classification of Federal Data on Race and Ethnicity, 
Notice of Decision." Federal Register 62 (30 October 1997): 58782-58790. 


"Final Guidance on Maintaining, Collecting, and Reporting Racial and Ethnic Data to the 
U.S. Department of Education, Final guidance." Federal Register 72 (19 October 2007): 
59266-59279. 


National Forum on Education Statistics, Race/Ethnicity Data Implementation Task Force. 
(2008). Managing an Identity Crisis: Forum Guide to Implementing New Federal Race and 
Ethnicity Categories (NFES 2008-802). National Center for Education Statistics, Institute of 
Education Sciences, U.S. Department of Education. Washington, DC. 


http://edpubs.ed.gov/productcatalog.aspx). 


Applicable Population: All learners. 

Federal Reporting: Total number of learners by racial/ethnic group is reported. 
Demographic Measure #2: Gender 

Definition: Whether the learner is male or female. 

Applicable Population: All learners. 


Federal Reporting: Total number of learners by gender is reported. 
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Demographic Measure #3: Age 
Definition: Years since learner’s date of birth. 
Applicable Population: All learners. 


Federal Reporting: Total number of learners by age is reported using the following age 
categories: 16—18 years, 19-24 years, 25—44 years, 45-59 years, and 60 years and older. 


Student Status Measure #1: Labor Force Status 


Definition: Whether the learner is employed, not employed, or not in the labor force at time 
of entry into the adult education program, according to the following criteria: 


e Employed—Learners who work as paid employees, work at their own business or farm, or 
who work 15 hours or more per week as unpaid workers at a farm or business operated by a 
member of their family. Also included are learners who are not currently working but who 
have jobs or businesses from which they are temporarily absent. 


e Unemployed—Learners who are not employed but are seeking employment, make specific 
efforts to find a job, and are available for work. 


e Not in the Labor Force—Leamers who are not employed and are not seeking employment. 

Applicable Population: All learners. 

Federal Reporting: Total number of learners by category is reported. 

Student Status Measure #2: Public Assistance Status 

Definition: Learner is receiving financial assistance from Federal, State, or local 
government agencies, including Temporary Assistance for Needy Families (TANF) or equivalent 
general assistance, food stamps, refugee cash assistance, old-age assistance, and aid to the blind or 
totally disabled. Social Security benefits, unemployment insurance, and employment-funded 
disability are not included in this definition. 


Applicable Population: All learners. 


Federal Reporting: Total number of learners receiving assistance is reported. 


Student Status Measure #3: Disability Status 

Definition: Learner has a record of, or is regarded as having any type of physical or mental 
impairment, including a learning disability that substantially limits or restricts one or more major life 
activities (e.g., walking, seeing, hearing, speaking, learning, and working). 


Applicable Population: All learners. 


Federal Reporting: Total number of disabled learners is reported. 
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Student Status Measure #4: Rural Residency Status 

Definition: Learner resides in a rural area; that is, a place with a population of less than 
2,500 that is not near any metropolitan area with a population greater than 50,000, or in a city with 
adjacent areas of high density. 

Applicable Population: All learners. 


Federal Reporting: Total number of learners living in rural areas is reported. 


Student Status Measure #5: Highest Degree or Level of School 
Completed 


Definition: The highest number of years of formal schooling the learner has completed or 
the highest credential or degree the learner has achieved. Schooling in the U.S. or abroad is included. 


Applicable Population: All learners. 


Federal Reporting: Total number of learners completing the highest grade level or 
credential is reported for schooling either in the U.S. or abroad. 


Teacher Status Measure #1: Total Years of Adult Education Teaching 
Experience 


Definition: The total number of years a teacher has taught in adult education. 

Applicable Population: All paid teachers. 

Federal Reporting: Total number of teachers by years of experience teaching adult 
education is reported, separately for full- and part-time paid teachers. 

Teacher Status Measure #2: Teacher Certification 


Definition: The credential or equivalent education achieved by the teacher, as defined in the 
following categories: 


e Adult Education Certification—a credential recognized by the State that focuses on 
teaching adult education students. 


e K-12 Certification— a credential recognized by the State that focuses on teaching 
children. 


e Special Education Certification— a credential recognized by the State that focuses on 
teaching children or adults with disabilities or special needs. 


e TESOL Certification— a credential recognized by the State that focuses on teaching 
English to speakers of other languages. 
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Applicable Population: All paid teachers. 


Federal Reporting: Total number of teachers by type of credential is reported, separately 
for full- and part-time paid teachers. 


Student Participation Measures 


Student Participation Measure #1: Contact Hours 


Definition: Hours of instruction or instructional activity the learner receives from the 
program. Instructional activity includes any program-sponsored activity designed to promote student 
learning in the program curriculum, such as classroom instruction, assessment, tutoring, or 
participation in a learning lab. 


Applicable Population: All learners. 
Federal Reporting: Total number of hours is reported. 
Measuring Contact Hours for Learners in Distance Education 


Students in distance education (defined below under Student Participation Measure #2) must 
have at least 12 hours of contact with the program before they can be counted for federal reporting 
purposes. Contact hours for distance learners can be a combination of actual contact and contact 
through telephone, video, teleconference or online communication, where student and program staff 
can interact and through which learner identity is verifiable. 


Optional Reporting of Proxy Contact Hours 


States may, but are not required, to report proxy hours of time students spent on distance 
learning activities. States providing distance education that want to measure and report proxy contact 
hours for these students must develop a state distance education policy that describes the following. 


e The curricula that local programs can use to provide distance education; 


e The model or models used to assign proxy contact hours for each type of curriculum. 
States must develop proxy contact hours using one of the following models.” 


Oo Clock Time Model, which assigns contact hours based on the elapsed time that a 
learner is connected to, or engaged in an online or stand alone software program that 
tracks time. 


“ See Project Ideal (2005), Working Paper No. 2 Measuring Contact Hours and Learner Progress in Distance Education 
Programs, Institute for Social Research, University of Michigan, for further information on the use and development of these 
models. 
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O Teacher Verification Model, which assigns a fixed number of hours of credit for 
each assignment based on teacher determination of the extent to which a learner 
engaged in, or completed, the assignment. 


Oo Learner Mastery Model, which assigns a fixed number of hours of credit based on 
the learner passing a test on the content of each lesson. Learners work with the 
curriculum and materials and when they feel they have mastered the material, take a 
test. A high percentage of correct answers (typically 70%-80%) earns the credit 
hours attached to the material. 


e The proxy contact hours assigned for completing requirements for each type of 
curriculum used (teacher verification model) or the proxy contact hours assigned for 
completion of units of material comprising the curriculum (learner mastery model). The 
state must use the proxy contact hour model appropriate for the distance education 
curricula. The state may use the clock time model with curricula that track time student 
spends interacting with instructional material and disconnects after a preset period of 
inactivity; and must describe the procedures the state used to develop proxy contact 
hours. 


Student Participation Measure #2: Program Enrollment Type 

Definition: Learner is enrolled in the following programs or institutions: 

e Adult Basic Education Program—A program of instruction designed for adults who 
lack competence in reading, writing, speaking, problem solving, or computation at a level 
necessary to function in society, on a job, or in the family. 

e Adult Secondary Education Program—A program of instruction designed for adults 
who have some literacy skills and can function in everyday life but who are not proficient 


or do not have a certificate of graduation or its equivalent from a secondary school. 


e EL Program—A program of instruction designed to help adults with limited English 
proficiency achieve competence in the English language. 


e Correctional Education Program—A program of ABE, ASE, or EL instruction for 
adult criminal offenders in correctional institutions. 


e Family Literacy Program—A program with a literacy component for parents and 
children or other intergenerational literacy components. 


e Workplace Literacy Program—A program designed to improve the productivity of the 
workforce through improvement of literacy skills needed in the workplace by: 


oO Providing adult literacy and other basic skill services and activities, including basic 
computer literacy skills. 


oO Providing adult secondary education services and activities that may lead to the 
completion of a high school diploma or its equivalent. 
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O Meeting the literacy needs of adults with limited English proficiency. 


e Program for the Homeless—A program designed for homeless adults. Homeless adults 
lack a fixed, regular, nighttime residence or have a residence that is (1) a publicly supervised 
or privately operated shelter designed to provide temporary living accommodations 
(including welfare hotels, congregate shelters, and transitional housing for the mentally ill), 
(2) an institution that provides temporary residence for individuals intended to be 
institutionalized, or (3) a public or private place not designed for, or ordinarily used as, a 
regular sleeping accommodation for human beings. The term homeless adult does not apply 
to any individual imprisoned or otherwise detained pursuant to an act of the Congress or a 
State law. 


e Correctional Facilities—Any prison, jail reformatory, work farm, detention center, or any 
other Federal, State, or local institution designed for the confinement or rehabilitation of 
criminal offenders. 


e Community Corrections Programs—A community-based rehabilitation facility or halfway 
house. 


e Other Institutional Programs—Any other medical or special institution. 


e Distance Education—Formal learning activity where students and instructors are separated 
by geography, time or both for the majority of the instructional period. Distance learning 
materials are delivered through a variety of media including, but not limited to, print, audio 
recording, videotape, broadcasts, computer software, web-based programs and other online 
technology. Teachers support distance learners through communication via mail, telephone, 
e-mail or online technologies and software. 


Note: For participants who receive both distance education and traditional classroom 
instruction during a program year (such as through a blended distance-classroom 
approach or concurrent enrollment in both types of instruction), the state must have a 
policy, consistent with the NRS definition, that defines how local programs are to classify 
the student. For NRS reporting, states can count a student only once, as either a distance 
education student or traditional classroom learner. 


Applicable Population: All learners. 


Federal Reporting: Total number of learners in each program or category is reported. The 
number of learners in each program type can be used to analyze the performance of these participants 
separately from the overall adult education population. 


SECONDARY STUDENT STATUS AND OUTCOME MEASURES 
(OPTIONAL) 


The NRS secondary measures are optional measures of student status and outcomes that 
States are not required to collect and that are not used as a basis for assessing State performance 
under WIA. The NRS includes these measures because many stakeholders during the consensus 
building process believed that these measures would be important to the goals and purposes of adult 
education. 
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Secondary student status measures of low income, displaced homemaker, and single parent 
status are included, because these groups are specific target populations under WIA. States that are 
required to report their services to these populations can use these measures, which are identically 
defined by the U.S. Department of Labor. There also is a secondary status measure to identify 
learning-disabled adults to assist programs in reaching these students. 


The secondary outcome measures are in the areas of employment, community, and family. 
The employment measure is whether the student’s public assistance grant is reduced or eliminated 
due to employment. This measure applies only to students receiving public assistance upon entry 
and can be collected through data matching or survey methods. In the area of community, there are 
three measures covering citizenship, voting, and community involvement. For students enrolled in 
citizenship programs, there is a measure of whether the student achieves the skills to pass the 
citizenship exam. Voting for the first time or registering to vote and more involvement in 
community groups or activities are the remaining measures. The family measures include increased 
involvement in children’s literacy activities and in children’s education. Voting and family measures 
should be collected through survey methods or from direct reports of learners. 


Another optional measure for national reporting is whether a student completed a work-based 
project learner activity. Project learners are students enrolled in a class with 30 hours or less of 
scheduled instruction with a goal of teaching specific workplace-related literacy skills. On 
enrollment, the learner and the program determine the specific skills to be learned and the method to 
assess the attainment of the skills. The assessment must employ a standardized test or be a 
performance-based assessment with standardized scoring rubrics. Programs do not collect the core 
outcome measures on students designated as project learners, and these learners are counted 
separately. This measure is included in the NRS to allow States and programs to serve learners with 
a short-term learning need without having a detrimental effect on performance of the core outcome 
measures. 


Optional Student Status Measures 
The following five optional student status measures target special populations identified 


under WIA. Information should be obtained through observation, learner self-report, or appropriate 
documentation on whether any status applies to learners. 


Secondary Student Status Measure #1: Low-Income Status 

Definition: The learner receives or is a member of a family who receives a total family 
income in the 6 months prior to enrollment of 70 percent of the income level standard for a family of 
that size, or the learner is receiving or is a member of a family who is receiving cash assistance 
payments from Federal or State agencies or food stamps, or the learner can be designated as 
homeless under the McKinney Act. 

Applicable Population: All learners. 


Federal Reporting: Total number of low-income learners is reported. 


Secondary Student Status Measure #2: Displaced Homemaker 
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Definition: Learner has been providing unpaid services to family members in the home, has 
been dependent on the income of another family member but is no longer supported by that income, 
and is unemployed or underemployed and experiencing difficulty obtaining or upgrading 
employment. 

Applicable Population: All learners. 

Federal Reporting: Total number of displaced homemakers is reported. 

Secondary Student Status Measure #3: Single Parent Status 

Definition: Learner has sole custodial support of one or more dependent children. 

Applicable Population: All learners. 

Federal Reporting: Total number of single parents is reported. 


Secondary Student Status Measure #4: Dislocated Worker 


Definition: An individual who receives an individual notice of pending or actual layoff from 
a job, or an individual who receives a publicly announced notice of pending or actual layoff. 


Applicable Population: All learners. 


Federal Reporting: Total number of dislocated workers is reported. 

Secondary Student Status Measure #5: Learning-Disabled Adult 

Definition: Learner with an IQ in the low-average and above level (70+ to any level) who 
has deficits (related to neurological impairments) in capacity in defined limited learning areas; this 
can include dyslexia (reading disability), dysgraphia (writing disability), and dyscalculia (math 
disability). The learner also has a history of previous educational efforts. 

Applicable Population: All learners. 

Federal Reporting: Total number of learning-disabled adults is reported. 


Secondary Student Status Measure #6: Learner Goals for Attending 


Definition: Learner’s reasons for attending the class or program, as defined in the following 
categories: 


e Obtain a Job—Obtain full- or part-time paid employment. 
e Retain Current Job—Upgrade skills to enable retention of current job. 
e Obtain a Secondary School Credential or Its Equivalent—Achieve sufficient skills and 


credit hours to earn a State-accredited secondary diploma or pass a State recognized high 
school equivalency test. 
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e Enter Postsecondary Education or Job Training—Achieve skills to enable enrollment in a 
postsecondary education program or job training program. 


e Improve Basic Literacy Skills—Improve overall basic literacy skills. 


e Improve English Language Skills—Improve overall skills in the English language (e.g., 
speaking, reading, and writing). 


e Obtain Citizenship Skills—Obtain skills to pass the U.S. citizenship test. 
e Achieve Work-Based Project Learner Goals—Obtain the skills needed to complete a 
project learner activity (i.e., a course of 12—30 hours duration designed to teach specific 


workplace skills). 


e Other Personal Goals—Any other goal related to instruction with a clearly definable 
outcome, such as passing a driver’s test or improving reading ability. 


Applicable Population: All learners. 


Federal Reporting: Total number of learners for each type of goal is reported. 
Secondary Outcome Measures 


Secondary Employment Outcome Measure: Reduction in Receipt of 
Public Assistance 


Definition: Learner’s Temporary Assistance for Needy Families (TANF) Grant or 
equivalent public assistance is reduced or eliminated due to employment or increased income. 


Applicable Population: Learners who are receiving a TANF Grant or equivalent public 
assistance at the time of enrollment in the program. 


Federal Reporting: Total number of learners whose grant is reduced or eliminated is 
reported, and a rate or percentage can be computed by dividing this total by the total relevant 
population (number of learners on public assistance at program entry). Grant reduction may be 
reported at any time during the program year. 
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Secondary Community Measure #1: Achieved Citizenship Skills 
Definition: Learner attains the skills needed to pass the U.S. citizenship exam. 
Applicable Population: All learners with a goal of obtaining citizenship skills. 

Federal Reporting: Total number of learners who obtain skills to pass the citizenship exam 
is reported. A proportion or rate can be computed by dividing this total by the total relevant 
population (number of learners who enrolled in citizenship classes or who had a goal of citizenship). 

Additional Guidance on Achieved Citizenship Skills Measure 

This measure is included to document learning gains of students who are enrolled in classes 
designed to give them the literacy skills and substantive knowledge to pass the citizenship exam. 
These students should have “obtain citizenship skills” designated as their goal for attending. To 
determine whether students achieve these skills, program staff should administer a State-approved 
test that measures the relevant skill areas—such as a practice citizenship test, sample forms, and 
speaking tests—at the conclusion of the citizenship class. If this measure is to be reported, it is the 


State’s responsibility to ensure that programs use an appropriate test, establish the standards for 
passing this test, and train and monitor local staff in its use. 


Secondary Community Measure #2: Voting Behavior 


Definition: Learner registers to vote or votes for the first time anytime during the program 
year. 


Applicable Population: All learners who at time of enrollment, are not registered to vote or 
who have never voted. 


Federal Reporting: Total number of learners who register to vote or vote for the first time 
is reported. 


Secondary Community Measure #3: General Involvement in 
Community Activities 


Definition: Learner increases involvement in the following community activities: 
e Attending or organizing meetings of neighborhood, community, or political organizations. 
e Volunteering to work for such organizations. 
e Contributing to the support of such organizations. 
e Volunteering to work on community improvement activities. 


Applicable Population: All learners. 
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Federal Reporting: Total number of learners who increase community involvement in any 
measure is reported. 


Secondary Family Measure #1: Involvement in Children’s Education 


Definition: Learner increases involvement in the education of dependent children under his 
or her care, including: 


e Helping children more frequently with their school work. 
e Increasing contact with children’s teachers to discuss children’s education. 


e Having more involvement in children’s school, such as attending school activities and parent 
meetings and volunteering to work on school projects. 


Applicable Population: All learners enrolled in programs that include a focus on family 
literacy. 


Federal Reporting: Total number of learners who increase involvement in any area is 


reported. A rate or percentage can be computed by dividing this total by the total relevant population 
(number of learners in programs that include a family literacy focus). 


Secondary Family Measure #2: Involvement in Children’s Literacy- 
Related Activities 


Definition: Learner increases involvement in the literacy-related activities of dependent 
children under his or her care, including: 


e Reading to children. 

e Visiting a library. 

e Purchasing books or magazines for children. 
Applicable Population: All learners enrolled in programs that include a focus on family literacy. 
Federal Reporting: Total number of learners who increase involvement in any area is reported. A 


rate or percentage can be computed by dividing this total by the total relevant population (number of 
learners in programs that include a family literacy focus). 


Work-Based Project Learner Outcome Measure: Completed Work- 
Based Project Learner Activity 


Definition: Learner acquires the skills taught in a short-term learning course designed to 
teach specific work-based skills. A short-term course is an instructional program of at least 12 hours 
but no more than 30 hours duration. 


Applicable Population: Learners enrolled in a short-term course and designated at entry as 
work-based project learners. 
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Federal Reporting: Total number of learners who complete a work-based project learner 
activity is recorded. A rate or percentage can be computed by dividing this total by the total relevant 
population (number of work-based project learners). Project learners are not counted for the 
educational gain measure and are not assigned an educational functioning level. No core outcome 
measures are reported for project learners. 


Additional Guidance on Work-Based Project Learners Measure 


Work-based project learners are enrolled in an instructional or training course that has at least 
12 hours and no more than 30 hours of scheduled instruction. The course must be designed to teach 
work-based literacy skills. The skills the student learns, and the method for assessing these skills and 
standards for achievement, must be explicitly stated prior to beginning the course. To be recognized 
as completing the activity, the learner must demonstrate achievement of the skills at the level of the 
agreed-upon standard. As with other student assessments of the NRS, the assessment must either be 
a standardized test or performance-based assessment with standardized scoring rubrics. It is the 
State’s responsibility to establish and monitor the assessment process and train staff on the use of the 
assessment procedures. 


Work-based project learners should designate “achieve work-based project learner goals” as 
their goal for attending. Once a student is designated as a work-based project learner, the student is 
not assigned an educational functioning level and no additional outcomes are collected on that 
learner. The learner is reported on the NRS reporting tables (in the optional secondary outcomes 
table). 


Work-based project learning should not be confused with workplace literacy programs, 
which also are designed to teach workplace skills. Workplace literacy programs have a longer 
duration, are open ended, and generally teach a broader range of literacy skills (see definition). 
Students enrolled in workplace literacy programs are counted under the required core outcome 
measures. 
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The NRS produces a set of measures that describes adult education students, their 
participation, and the outcomes that they achieve. These measures are used at the State and national 
levels to demonstrate whom the adult education program serves and its impact on learners’ 
educational and employment-related outcomes. At the local level, programs collect data and train 
staff according to policies and procedures set by the State for program management and 
improvement activities and to report on performance. This chapter describes the flow of data from 
the local programs through States to the ED. It also summarizes the roles and responsibilities of 
local programs and States in relation to their specific data collection processes and to the operation 
and maintenance of the NRS at the Federal level. 


THE NRS DATA FLOW FRAMEWORK 


The development of a national database for adult education requires the close collaboration 
among the ED, eligible State agencies (e.g., State education agencies, community college boards, 
departments of labor), and local programs. Each entity has an essential role in the operation and 
maintenance of the system that helps ensure the collection of valid and reliable data from programs 
and States. 


e At the Federal level, ED supports a national database for adult education by developing the 
framework and measures for the NRS. The Federal role is to establish NRS measures, 
methods, and reporting requirements; ensure valid and reliable data; provide assistance to 
States in understanding and implementing these requirements; negotiate performance levels 
with States; monitor the system to ensure that it is producing valid and reliable measures; 
report the data to Federal agencies; decide on State incentive awards based on NRS data; and 
maintain the national database of measures. 


e States are responsible for implementing NRS measures, methods, and requirements in a way 
that meets Federal guidelines; setting State performance standards; providing resources, 
training, and support for data collection to local programs; monitoring local programs using 
quality control procedures to ensure data validity; maintaining a database that includes data 
from all local programs; establishing a written policy for collecting follow-up measures; and 
implementing data matching procedures when data matching is used as the follow-up 
methodology. In addition, States must have a written assessment policy to ensure that 
measures of educational gains are meaningful by establishing a standardized assessment 
system based on tests or authentic performance. States are to use NRS measures to promote 
continuous improvement based, in part, on their performance on NRS measures. 


e Local programs are responsible for allocating sufficient resources to collect NRS measures 
and reporting them according to State requirements. Local programs have primary 
responsibility for collecting these measures using valid, uniform procedures to ensure 
comparability among programs and must maintain these data in an individual student record 
system. To ensure that educational gains are standardized, programs must have common 
methods for assessing students at intake and following instruction. In States using the survey 
follow-up methodology, programs also must conduct a follow-up survey on students. 
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Exhibit 3.1 shows the general data flow framework envisioned for the NRS by following the 
movement of data at each of these three levels (Federal, State, and local). At the local program level, 
each of the program’s instructional sites collects measures from students at three time periods, 
including intake, update, and follow-up. Upon a student’s intake into the program, local staff collects 
descriptive measures—demographic information, student status measures—and conduct an 
assessment of the student’s educational functioning level for placement. During the course of 
instruction, program staff (typically teachers) provides at least two additional measures about the 
student: contact hours or attendance and a progress assessment. The progress assessment is usually 
administered at a time established by State policy and may be at the end of the course of instruction, 
at the end of the program year, or after a set number of instructional hours. 


In States using the survey methodology, local programs also are required to collect the core 
follow-up measures on students. These measures include employment-related measures, Measures on 
placement in postsecondary education or training, and obtainment of a secondary school credential or 
its equivalent. NRS follow-up procedures require program staff to collect the employment measures 
through a student survey in the first and third quarters after students’ exit quarters or by matching 
procedures. In States that use the data matching methodology, the collection of follow-up data 
becomes a State responsibility. 


Local programs must combine all of the measures collected at each instructional site into an 
individual student record system. This type of system is essential to the NRS, because it allows local 
programs to conduct analyses of outcomes for specific student groups for reporting and program 
management. For example, only an individual record system allows analysis of such issues as 
whether specific types of students achieved specific outcomes or the number of instructional hours 
needed by groups of students to advance an educational level. The NRS does not specify the 
software or design of the student record system and leaves this to local and State discretion. 


Exhibit 3.1 also shows the movement of NRS data from the local program to the State level. 
Each local program must submit its data to the State education agency to enable the State to develop 
a statewide adult education database. At the end of the program year, States must submit data in 
aggregated data tables to ED, which maintains a national database. This submission is required to be 
in electronic form, using software developed for this purpose by ED. 


Data Collection: The Federal Role 


At the Federal level, ED’s role is to establish the NRS procedures through an inclusive 
process that addresses Federal legislation, responds to State and local concerns, and coordinates with 
Federal partner agencies. ED developed and pilot tested the methodology and definitions for the 
reporting system and produced corresponding guidelines. In addition, ED monitors the 
implementation of the NRS, conducts quality control of State procedures, and provides ongoing 
technical assistance and training to States. This training supports State efforts to train staff and 
implement the NRS to produce valid, uniform, and reliable data. Learning to use data more 
effectively for program improvement and accountability is another focus of training as a means to 
enhance the value of the NRS and to encourage adoption and support of the system. Technical 
assistance materials are provided to States on issues such as local program quality control, 
assessment procedures, and program monitoring. 
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Site Procedures 


Intake 
* Teacher 
* Student 
+ Intake Staff 


Update 
* Teacher 
+ Program Staff 


Followup 
+ Program Staff 
* Teacher 


Descriptive Measures 
* Demographics 
* Goals 


* Status 


Educational Gain Outcome 
* Initial assessment 


Participation Measures 

* Contact Hours 
Educational Gain Outcome 
+ Progress assessment 


Followup Outcome Measures 
* Placement in secondary 
education or training 


* Obtained secondary 
credential 

+ Entered or retained 
employment 


Exhibit 3.1 
National Reporting System Data Flow Framework 
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* Reporting 
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+ Support program development 
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System Support 
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* Other analyses 


System Support 

+ Enforce policy/regulation 

* State performance 
standards 

* Training and technical 
assistance to states 

* Quality control for 
reliable and valid data 
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All States are required to submit their aggregate data to ED annually using NRS data tables. 
ED then creates a national report and submits this report to the U.S. Congress and other audiences. 
Prior to creating the national report, ED reviews each State’s data tables for errors and 
inconsistencies and asks for corrected data tables from States, as needed. In turn, States may need to 
review again local program data to correct data problems and contact local program directors for 
corrections. Local staff then needs to identify problems, correct errors, and resubmit data to the 
State, which then provides corrected tables to ED. ED uses the data collected from States to 
determine State performance incentives. 


Data Collection: The State Role 


The NRS requirements present a common framework that provides standards and consistency 
for national data collection. However, States have the responsibility for developing policies and 
implementing procedures that meet NRS requirements and work within the State’s environment and 
delivery system to produce valid and reliable data. 


Assessment Policy 


One critical area where the Federal—State interface occurs is in the measurement of learning gains 
within the educational functioning levels. To accommodate State variation in instructional emphasis, 
goals, and assessment polices, the NRS allows States to establish their own procedures for student 
placement and assessment to measure learning gains. Each State must have an assessment policy 
that describes the assessments local programs may use and the timeframe for pre- and posttesting 
students. The assessments may be standardized tests or an alternative assessment if the assessment 
meets accepted psychometric standards for valid and reliable assessment, including empirically 
validated scoring rubrics with high inter-rater reliability. Programs may use only assessments that 
have been approved by OCTAE for measuring educational gain within the NRS framework. OCTAE 
conducts the approval process annually using panels of independent experts in assessment who 
evaluate assessments according to the process outlined in 34 CFR Part 462 (see Federal Register, 
Vol. 73, No. 9, January, 14, 2008). 


Unvalidated rubrics and checklists and locally developed tests do not meet these criteria and 
are not acceptable. However, it is acceptable for a State to have more than one assessment, such as 
one test for ESL students and a different test for ABE students, as long as there are clear procedures 
for when to use each test. The State policy also should designate when programs should pretest 
students and the calendar time or instructional hours when programs should posttest students. The 
policy also should clearly state that programs are to use a different form of the same assessment for 
pre- and posttesting. Chapter II of this document presents greater details of these requirements. 


Follow-up Methodology 


The State must determine a methodology for collecting the NRS follow-up measures of 
entered employment, retained employment, attainment of a secondary credential, and entry into 
postsecondary education. States must use data matching, a follow-up survey, or a combination of 
these methods to collect these measures. For example, a State may use the survey for employment 
measures and data matching to determine which students passed a State recognized high school 
equivalency test. The survey must include all local programs, although the State or a third party may 
conduct the survey. See Chapter II for more information on these requirements. 
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Secondary (Optional) Measures 


If a State decides to collect any or all of the NRS optional measures, State policy should 
clearly identify these measures and define them consistently with NRS definitions. The policy also 
should state the methodologies for collecting these optional measures, which may include survey, 
data matching, or direct reporting from students while they are enrolled. 


Data Reporting Timelines and Formats 


The State must have requirements for local programs to report data according to a fixed, 
regular schedule. Programs submit data to a central source, such as the State or district, according to 
this schedule. The reporting periods for local programs must be monthly or quarterly in order to 
minimize incomplete reports and potentially false data that result from longer time periods between 
report cycles. Another reason for frequent reporting is that errors or problems may be identified and 
corrected on an ongoing basis. If data are reported only once or twice a year, it is not possible to 
identify errors before it is too late to correct them. The State also should specify the technical format 
in which data are to be submitted so that it is consistent with State reporting software. 


A System of Quality Control 


To verify the validity of data and to ensure local program compliance with State data 
collection policies, the State should conduct frequent reviews of data immediately after local 
programs submit them. Monitoring procedures also should include regular discussions with local 
data collection staff, either at State meetings or by telephone and e-mail, to discuss problems. To be 
most effective, monitoring should be proactive, nonpunitive, and presented as a form of technical 
assistance. With this approach, local staff is less likely to hide problems and cover up mistakes. 
Monitoring also should include at least occasional onsite auditing of data. Quality control is 
described in more detail in Chapter IV. 


The WIA requires States to evaluate local program performance on NRS core measures as 
one condition of local funding. States may use any other indicators of their choosing in evaluating 
programs and making funding decisions. The NRS core measures do not limit or preclude use of 
other measures. Indeed, the inclusion of secondary, optional measures in the NRS framework is 
intended to provide States with additional options on such measures. For example, States that wish 
to place a greater emphasis on family literacy or community involvement could include the 
secondary measures in these areas in their evaluation of local programs and fund them according to 
performance on these measures. Similarly, States wanting to emphasize serving students on public 
assistance could use the measure of welfare reduction in local performance evaluations. 


Software or Technical Standards for Local Data Collection and 
Reporting 


To meet NRS reporting requirements, the State must have software that is capable of 
aggregating NRS data from all local programs and producing the required data tables for Federal 
reporting. To report data to the State, local programs must have an individual student record 
database in a relational format. Each State must establish a State database system for local programs 
or provide programs with uniform technical standards for database development to allow State 
reporting. All software should have the ability to produce “edit reports” and possess error checking 
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capabilities to identify missing and inconsistent data. These requirements for data collection are the 
minimum—additional data and reporting from local programs may be required to meet needs of the 
State. 


Ongoing Training and Technical Assistance to Local Programs 


Because local adult education program staff collects NRS data, they must fully understand 
policies and procedures if they are to produce quality data. Thus, it is critical to the success of the 
NRS that States provide training to teachers and other local staff involved in collecting and reporting 
data. This training should be ongoing so that training is available for new staff. Critical topics for 
training include definitions of measures, completing reporting forms, conducting assessments, and 
follow-up methods. While training should cover the general procedures and methods of the NRS, 
additional training on the importance of data and how to use it is likely to increase data quality. 
When local staff can see how to use data for their own purposes, their data collection activities 
become more meaningful and they are likely to take more care in collecting data. 


Exhibit 3.2 presents a summary of the policies and procedures that States must have in place 
for the NRS. 


Exhibit 3.2 
Summary: State NRS Policies and Procedures 


Data Collection: The Local Role 


Local programs are on the front lines of the data collection system and they must allocate 
sufficient resources, including both staff and funds, to collect information from students—the 
descriptive, participative, and outcome measures that comprise the database. For these data to be 
meaningful on a statewide and national basis, data collection procedures must be standardized among 
all programs in each State; that is, the data must be defined and collected in the same way by all 
programs to make it comparable. The role of local programs is central to data collection efforts. To 
achieve standardization of data collection, program staff needs ongoing training and assistance in: 


e Understanding the definitions of each measure and having clear guidelines on how to record 
these measures, including how to handle missing or incomplete data. 


e Understanding of and compliance with the State-defined procedures for assessing students 
for placement into educational functioning levels and assessing progress. 


e Following procedures for implementing the follow-up survey, if it is conducted by the 
program. 
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e Understanding how to correctly record and report data to the State. 


It is the State’s responsibility to provide training and technical assistance to local programs to 
achieve these competencies. 


With adequate resources and proper training, local program staff collects data to report to the 
State. This collection process must produce reliable and valid data in order to be useful to the 
programs and the State. Data are reliable to the extent that they are collected in the same way, by 
different people, and at different times. In other words, no matter who collects the data or when data 
are collected, the same data collection procedures are consistently implemented in the same way. 
Data are valid only to the extent that they represent what they are intended to represent. For 
example, if the program reports that 40 percent of students have gained a level according to test 
scores, those test scores (if they are valid) will accurately convey the score and interpretation 
intended by the test’s publisher. 


There are three components to collecting valid and reliable data: (1) a well-planned, effective 
process; (2) resources to implement the process; and (3) clearly defined procedures for collecting 
each measure. Next is a discussion about the data collection process that contains these three 
components and a method for evaluating the process. 


Model Data Collection Process 


Like other processes, data collection requires planning, constant attention, oversight, and 
fine-tuning through monitoring, error checking, and training. With a sound, well-planned process, 
sufficient resources, and oversight, the program can have a data collection system that produces valid 
and reliable data to assist program management and promote improvement. 


The specifics of individual approaches to data collection vary among programs, but Exhibit 
3.3 presents a model data collection process, starting with student intake and tracing the process to 
the end goal—submission of State and Federal reports. This model illustrates the key components of 
a good data collection system and staff roles at each step. A discussion of the key components 
follows. 


Intake 


When students enter the program, intake staff collect NRS measures, including age, ethnicity, 
race, and gender. Intake often includes a goal-setting process where students, with staff guidance, 
decide on short- and long-term goals for attending class. If the program uses a follow-up survey, 
then the process should include procedures for informing students that they may be contacted after 
they leave class. Intake staff completes an intake form and send the form to clerical staff and/or 
teachers. 
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Exhibit 3.3 
Local Data Collection: A Model 
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Teachers 


Teachers have a large role in data collection in most programs. Teachers must report student 
attendance or contact time, assess students, report test scores, and help students set goals. In 
addition, teachers who have direct contact with students are often asked to provide student 
information that was missing or incorrect at other stages of the data collection process. Teachers 
complete forms and, ideally, have a role in reviewing data and reports. 


Clerical and Error Checking Staff 


The data collection process results in a high volume of paper—forms, test scores, attendance 
records, and surveys—that clerical staff receive and track. Clerical staff must develop an organized 
system for managing this paper flow that includes receiving forms from other staff for checking and 
correcting. After error checkers correct forms, clerical staff then submits forms for data entry. 


Data Entry and Data System 


One or more staff must enter information into the program’s database. Data entry may occur 
at an instructional site, or the program may have a central data entry point to which all sites submit 
their forms for entry. Programs should have an individualized student database that is organized to 
allow the program to examine relationships among student and program variables, attendance, and 
student outcomes. After forms are keyed, data entry staff should review error reports promptly and 
resolve errors and missing data by returning forms to the staff members who collected the problem 
data. 


Reporting and Error Checking 


An essential feature of the data collection process is regular and frequent review of data 
entered into the data system. The data system should have preprogrammed error reports that allow 
for a review of inconsistent, out-of-range, and missing data. Data entry and clerical staff should 
regularly review these reports and should return them to teachers, intake workers, and clerical staff to 
clarify problems and obtain the missing data. Corrections should then be sent to data entry staff for 
entry into the database. 


Program Administrative Review 


The process should include a regular opportunity for the program director and other program 
leaders to review data reports. As the person most responsible, the director may often be the only 
person in the program who can see the big picture and thus, brings a different perspective to the data 
review process. This review may raise further questions about data integrity, requiring another round 
of data checking and verification among the staff. The program director may share data reports with 
staff as a means to identify problems, track progress, and receive staff buy-in into the data collection 
process by demonstrating how data can be used for program management and improvement. 


Local Data Collection Policies and Procedures 


In addition to following a clear model of data collection, local programs must establish 
policies and procedures for data collection that comply with State NRS requirements. This section 
presents the policies and procedures that local programs need to have in place. 
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Staff Roles and Responsibilities for Data Collection 


Every staff member in an adult education program plays a role in the data collection process. 
Intake staff collects student demographic data, teachers report attendance and may administer tests 
and report other outcomes, administrators must review and make decisions based on data tables, and 
administrative staff may be involved in checking forms and data entry. The State must ensure that 
every local program has clear written descriptions of the data collection process and the role of each 
individual in that process. In fact, local program job descriptions should incorporate the data 
collection responsibilities of the job, and performance reviews should consider how well staff fulfills 
these functions. 


Clear Definitions of Measures 


Local programs’ policies and procedures should include a written, precise definition for each 
data item that is compatible with the State definition. Some programs and States, for example, have 
a data dictionary that defines all measures and categories within measures. Although some measures 
may seem straightforward—ethnicity or sex, for example—others may require detailed explanation. 
Even seemingly simple definitions can sometimes require elaboration. For example, States should 
clarify the definition of how to classify the ethnicity of a student who claims to be part Asian and part 
White. Potential ambiguities show how helpful it is to customize definitions to the particular 
circumstances of State programs and to include examples of how to resolve ambiguities. 


Standard Forms for Collecting Data 


Staff must record information on intake and other data forms. Then, administrative staff keys 
the information from these forms into the program database. Consequently, the program should use 
standard forms for data collection that include all the data elements and categories that are referenced 
in the database system. Staff should not need nor be allowed to enter their own codes or variables, 
because this will cause data entry errors and hurt reliability and validity. Some States with uniform 
State database systems have standard forms used statewide for this reason. 


Error Checking and Quality Control Systems 


Data collection is a complex activity—mistakes and missing data are inevitable. For 
example, staff may fail to complete forms fully because of high workload or simple oversight, or the 
required information may not be available when it is needed. The data collection system must have 
procedures for checking data for completeness and accuracy at several points during the process. 
Data checking should follow a regular, prescribed schedule with clear deadlines. More than one staff 
person should be assigned to perform these data checking functions, and these functions should be 
made explicit in the staff job descriptions and throughout the program. Data checkers should review 
all data forms as soon as possible for completeness and accuracy and should receive error reports 
from the database to check immediately after data entry. To do their job, data checkers must have 
access to all staff—teachers, intake staff, counselors, and administrative staff—and the authority to 
obtain cooperation from them. 


Ongoing Training on Data Collection 
Staff must understand and follow data collection procedures to ensure valid and reliable data. 


To this end, training should be provided to staff to clarify their roles and responsibilities and to 
highlight the importance of data collection. The program should provide this training to all staff, and 
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training should be offered several times during the year, if possible, to accommodate new staff and to 
allow existing staff to take follow-up training. Regularly scheduled staff meetings or inservice 
trainings on data issues also provide staff with opportunities to discuss problems and issues that arise 
during data collection. Addressing these issues promptly helps the program avoid more serious data 
problems later. (More detailed guidance on data collection training for staff is provided later in this 
chapter.) 


Student-level, Relational Database System 


To use data for program improvement, staff must be able to look at outcomes and 
demographics for individual students according to such variables as the number of instructional hours 
received, length of time of enrollment, the teachers and classes enrolled, and the student’s 
educational functioning level. This type of analysis requires a database that stores information by 
individual students and links the different pieces of data for each student in reports or other output— 
a system known technically as a relational database. 


Clear and Timely Data Entry Procedures 


The procedures for data entry should specify at least one person whose job it is to enter the 
information from data collection forms into the program’s database. All staff members should know 
this person’s role, and he or she should have the authority to request clarification and to resolve 
errors. In addition, data entry should be scheduled at frequent, regular intervals, such as weekly or 
monthly. Without frequent data entry, the program may end up with a large backlog of forms to 
enter and staff may not become aware of errors and missing data on forms until it is too late to 
correct them. Part of the data entry procedures should also include a prompt, organized way to 
identify and resolve errors. For example, soon after data are entered, staff should be able to print out 
an error report for review. Staff should then use the error report to resolve missing data issues and 
correct errors as soon as possible after data entry. 


Timely or Direct Access to Database 


Local program staff members must have access to data for use in program improvement and 
management. The database system should have the capability for local program staff to access their 
data in useful ways. It is best if this access is direct, so that staff at the local level can query the 
database to print a report locally. Access through a third party or through the State also is useful if 
staff can request and receive data in a timely fashion. The usefulness of the data is limited when 
there is a great time lag between the request and receipt of data. 


Regular Data Reviews 


The program’s data collection procedures should include a regular data review by staff soon 
after entry into the database. Regular data reviews allow staff to identify errors, missing data, and 
other data that don’t make sense. Data reviews are also useful as a staff development opportunity to 
examine problems and issues in support of program improvement. Data can help staff understand 
issues such as the impact of instructional arrangements, learner retention, and learner progress. This 
will not only foster program improvement, but it may also improve data quality, as staff recognizes 
the importance of data collection to produce accurate and valuable information. 
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Exhibit 3.4 summarizes local program policies and procedures. 


Exhibit 3.4 
Summary: Local Program Data Collection Policies 
and Procedures 


Staff has a clear description and understanding of their roles and responsibilities for data collection. 


Clear definitions for each measure are established. 


Program uses standard forms, tied to the program database, for collecting data. 


Program has an error checking and quality control system for identifying missing and inaccurate data. 


Program has ongoing training on data collection. 


Program has a student-level, relational database system. 


Data entry procedures are clear and timely. 


Staff has timely or direct access to information from the database. 


Staff regularly reviews data. 


Local Staff Training Policies and Procedures 


To ensure that the data collected are of high quality, local programs should implement 
ongoing staff training on NRS procedures. Without training, staff will not know or understand the 
policies and procedures, and they will implement procedures in incomplete or haphazard ways that 
can impair data reliability and validity. 


The State should support training on data collection, and the local program also may provide 
training directly to staff on the specific procedures at each site. Staff training in data collection 
policies and procedures should include content on good professional development practices, as 
summarized below. 


Training on NRS Policy and Data Collection Procedures 


All program staff should be trained and fully knowledgeable of NRS policy, accountability 
policies specific to the State and locality, and the program’s data collection process. Training on data 
collection should cover each individual’s job in the process and include a review of others’ roles and 
how these roles and activities are connected, including the flow of data. (See exhibit 3.3 as a model 
for the flow of data collection at the local level.) The training must be specific and detailed, 
addressing such mundane topics as completing forms, data entry procedures, error checking, the 
program’s database system, and general accountability requirements. A schedule should be 
established to train new staff members and provide follow-up and ongoing training for existing staff. 


Continuous Professional Development on Data Collection 


One-shot trainings on any topic generally do not have lasting impact. Staff forgets 
procedures, misunderstand some part of the training, or think some procedures do not work 
effectively and do not follow them. A continuous system of professional development helps resolve 
these problems. Given the often high turnover among adult education staff, a continuous training 
protocol also gives local programs an ongoing mechanism to train new staff. Regularly scheduled 
trainings throughout the year that employ different modalities of training to improve effectiveness 
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and impact on data collection procedures are best. For example, the program might schedule general 
workshops, individual peer mentoring, shadowing, or project-based learning activities throughout the 
school year. 


Training Addresses Staff Needs 


Although all staff should receive initial general training on data collection, State 
accountability, and NRS requirements, the training should be designed according to the needs of 
local program staff. Using a periodic formal or informal needs assessment, collaborative planning 
process, or review of procedures, such as those outlined in this guide, the program can identify areas 
where staff need or want further training. Using this input to design training will make it more 
relevant to staff, thereby increasing interest and the likelihood that the training will result in 
improved data collection procedures. 


Use Effective Trainers and Methods 


The trainers who provide training to local staff are almost as important as the content of the 
training. Staff should have respect for the trainers who should be knowledgeable about the data 
collection process. Ideally, trainers are articulate, well organized, and encourage the contributions 
and input of participants in the training. In addition, the training is likely to be more effective if it 
employs interactive, hands-on activities, rather than just a lecture. An interactive training may 
involve asking staff to analyze actual data tables and then having them troubleshoot problems on 
their own. 


Training Results in Learning and Improved Practice 


The goal of professional development is to change staff behavior. For training on data 
collection, the desired outcome is that staff learns and then correctly follow all procedures. Although 
it is difficult to determine a cause—effect relationship between professional development, learning, 
and behavioral change, general patterns in data or observations of staff behavior at work may provide 
an indication of training effectiveness. For example, after training on assessment, staff may be 
observed as they administer tests or review student assessment records. Trainers may follow up by 
examining the assessment data produced by individual staff members to examine procedural 
differences. The professional development approach should include ways to verify staff learning and 
implementation of procedures that staff has learned. 


Exhibit 3.5 summarizes the local program training policies and procedures. 


Exhibit 3.5 
Summary: Local Staff Training Policies and Procedures 


e Staff receives training on NRS policy and data collection procedures. 
e — Asystem of continuous professional development on data collection is in place. 
e 


Training addresses staff needs. 


e Trainers effectively use interactive and hands-on activities to lead training. 


e Training results in learning and improved practice. 
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The data used for NRS are only useful if they are valid (i.e., measure what they are supposed 
to measure) and reliable (i.e., collected in the same way by different people and at different 
locations). To obtain valid and reliable data, data collectors at the State and local levels must 
understand the measures and follow the proper procedures for collecting the measures at all times 
with all people. States are responsible for promoting data quality and implementing training and 
quality control procedures for NRS measures. This chapter provides a brief overview of quality 
control methods that can be implemented prior to data collection, during data collection, and 
following data collection. The chapter also presents NRS reporting requirements, including student 
record software requirements and the required and optional NRS tables that States must submit 
annually to the Office of Vocational and Adult Education. 


DATA QUALITY CHECKLIST 


To allow the U.S. Department of Education to assess the quality of NRS data, States must 
complete the NRS data quality checklist. States are to submit this checklist with their annual NRS 
data submission. The checklist describes State NRS policies and the data collection procedures that 
local programs follow to collect NRS data. It provides an organized way for DAEL to understand 
and evaluate NRS data quality by defining data quality standards in four areas. State eligibility for 
incentive awards under WIA is contingent upon having superior quality ratings on the checklist. 


Data Foundation and Structure 


This content area addresses whether the State has in place the foundation and structures for 
collecting quality data that meet NRS guidelines. Standards measure whether the State has policies 
for assessment and follow up; whether local programs know these policies; and whether the State 
conducts validity studies to ensure processes are working to produce accurate and reliable data. 


Data Collection and Verification 


This area measures whether the State collects measures according to NRS guidelines using 
procedures that are likely to result in high reliability and validity. Standards also address whether 
data are collected in a timely manner, are systematically checked for errors, and whether the State 
also has processes for verifying the validity of the data. 


Data Analysis and Reporting 


The quality standards in this content area include whether the State has systems for analyzing 
and reporting data, including appropriate databases and software. The standards also address 
whether analyses and reports are produced regularly, are used to check for errors and missing data, 
meet NRS and State needs, and are useful to State and local staff for program management and 
improvement. 
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Staff Development 


The standards under this area address whether the State has systems for NRS professional 
development for State and local staff, including whether the State provides training on data 
collection, measures, assessment, and follow-up procedures. Standards also focus on whether the 
training is ongoing and continuous, meets the needs of State and local staff, and is designed to 
improve data quality. 


Levels of Quality and Quality Improvement 


Within each area there are three levels of quality that reflect whether the State has policies 
and procedures likely to improve the reliability and validity of data. Based on the checklist, DAEL 
classifies States’ NRS data procedures into one of these levels each year. To be eligible for incentive 
awards under WIA, States must meet standards at the superior level or higher. 


e Acceptable Quality. State policies and procedures for implementing the NRS meet the 
essential requirements for NRS implementation as described in the NRS Implementation 
Guidelines and the Guide for Improving NRS Data Quality. 


e Superior Quality. State procedures go beyond the minimum to promote higher levels of 
data validity and reliability through more rigorous definitions, regular oversight of data 
collection methods, ongoing assistance to local programs on NRS data issues, and procedures 
for verifying the accuracy of data. 


e Exemplary Quality. The State has procedures and systems that promote the highest levels 
of data validity and reliability, including systems for verifying data accuracy from local 
programs, systems for monitoring data collection and analyses, and corrective systems to 
improve data on an ongoing basis. State procedures indicate a focus on continuous 
improvement of the quality and accuracy of data. 


States have to meet all of the standards within a quality level to be considered at that level. 
In addition, the scoring is cumulative, so that to score at the superior level, a State has to meet all of 
the standards for that level and all standards for the acceptable quality level. To rank at the 
exemplary level, States have to meet all of the standards for all quality levels. Appendix B includes a 
copy of the checklist. 


IMPROVING DATA QUALITY 


The data quality checklist defines data quality in the NRS and provides guidance to States on 
how to improve quality. This section summarizes how States can improve quality in three ways: 
training local staff, improving local data collection, and local monitoring and data audits. 


Training 


Within the NRS, data collectors are local program staff. Therefore, it is critical to NRS’s 
success that teachers and other local staff involved in collecting and reporting data receive both 
preservice and inservice training on the NRS. Critical topics for training include definitions of 
measures, completing reporting forms, conducting assessments, and follow-up methods. 
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Understanding and correctly using State assessment procedures are critically important to 
NRS data quality, given the central importance of the educational gain measure. Accurate reporting 
of this measure requires local staff to implement the State assessment methods for intake and 
progress assessment. For example, progress assessment must be administered at the appropriate 
time, as determined by the State and staff, and must follow standardized procedures. Failure to 
follow the correct procedure for administering a standardized test invalidates the test results. 


Training should cover the general procedures and methods of the NRS, but additional 
training on the importance of data and how to use it is likely to increase data quality. When local 
staff can see how to use data for their own purposes, their data collection activities are more 
meaningful and they are likely to take more care in conducting them. 


Quality of data also is enhanced when resources are available, including State or other local 
staff to consult when questions or difficulties arise. Through the NRS, DAEL developed NRS 
Online (http://www.nrsonline.org), a Web-based training site, to assist States in training local staff. 
Training materials also are available on the NRS project Web site (http://www.nrsweb.org). 


Local Data Collection 


During the data collection process, States and local programs can implement four 
mechanisms to help ensure data quality. First, data collection procedures need to be explicitly 
organized. Program staff should establish specific, concrete procedures for data collection and data 
reporting. These procedures should state what is to be collected, when it is collected, and who is 
responsible for collecting it. The time when the information should be collected and reported also 
should be determined. Incorporating these procedures formally into staff job responsibilities 
enhances the likelihood that staff performs them. 


The second critical factor to collecting quality data is devoting sufficient resources—time, 
staff, and money—to data collection. Providing resources shows staff that data collection is a valued 
and important activity, not something that is done as an afterthought or when there is time. At least 
one staff member in a program should have explicit responsibility for ensuring data are collected and 
reported. 


Reporting data in a timely manner according to a fixed, regular schedule is the third factor for 
promoting data quality. Data should be reported to a central agency, such as the State or district, 
frequently and at fixed time periods. At the local level, information should be entered into the 
program’s MIS as frequently as possible. For example, attendance should be reported weekly or 
monthly. For reporting to the State, monthly or quarterly reporting is highly preferred. If the time 
lag for reporting data is too long, then the data is not reported completely, as staff has a tendency to 
put off data reporting until the deadline. The result is a high degree of missing and possibly false 
data. Another reason for frequent reporting is that errors or problems can be identified and corrected 
on an ongoing basis. If data are reported only once or twice a year, errors may go unnoticed before it 
is too late to correct them. 


Finally, frequent contact with data collection staff and spot checking their data assists in 
ensuring quality data. A State or local staff member knowledgeable in reporting and data collection 
should provide regular, ongoing monitoring of data collection through scheduled contact with local 
staff. Samples of data collection forms should be examined periodically. To be most effective, 
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monitoring should be proactive and nonpunitive and viewed as a form of technical assistance. With 
this approach, staff is less likely to try to hide problems or cover up mistakes. 


Local Monitoring: Data Reviews and Data Auditing 


One of the simplest ways to audit local programs is to review local data. A data review 
should examine disaggregated data from all local programs. Aggregated State data (i.e., summary 
data from all local programs combined) may mask important details and clues about what the data 
reflect. Types of data to examine include: 


e The number and percentage of students who are pre- and posttested by type of student and 
date of posttesting. 


e The percent of students who advance by level. 
e The number and percent of students who achieve follow-up outcomes. 


e Students’ average attendance hours and number of hours it took students to advance and 
achieve follow-up outcomes. 


Critical review of these data may identify patterns that raise questions or seem improbable 
with numbers that seem unrealistically high or low, for example. Data reviews also can be used to 
study local adherence to State policies and differences by types of students and programs. 


A more formal way to investigate local program adherence to State policies and to study data 
quality is to conduct a local program data audit. Like a financial audit, a data audit involves an onsite 
review of the actual data forms and files and verification of the accuracy and validity of the 
information on the forms. Often, an independent third party conducts the audit, such as an 
accounting firm or a compliance review agency from the State government. States should perform at 
least occasional data auditing of a sample of programs because this type of review is the most 
accurate way to assess data validity at the local level. Findings from the audit can help identify 
technical assistance and training needs and prevent future problems. 


The auditing process should include at least four procedures. First, the auditor should 
interview program staff involved in data collection regarding the procedures they follow, particularly 
regarding how staff deals with missing and incomplete information, data entry procedures, and 
reporting times. The auditor also should review the program’s assessment and follow-up procedures 
to ensure that they comply with State policy. 


Second, the auditor should examine a random sample of student records for completeness and 
accuracy. The sample size must be large enough to make inferences about the program overall and 
to accommodate the expected high percentage of students whom the auditor is unable to reach. The 
auditor should compare the written records and information on the selected students’ forms with 
information that is in the program’s MIS to ensure correspondence between the sources. This review 
informs the auditor about whether staff completes forms fully and accurately and whether there are 
problems transferring information from the forms to the program’s database. 


Next, the auditor should contact the sample of students by telephone to obtain verification 
on key variables such as: 
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e Attendance—Ask students to recall dates of active enrollment and approximate frequency of 
attendance. 


e Tests and assessments—Ask students to recall whether they took tests and assessments and 
when they took them, what goals they set, and why they attended classes 


e Outcomes for follow-up measures. 
e Satisfaction with services. 


To minimize interviewer bias, States should prepare a formal protocol and standard script for 
auditors to follow when making these calls. 


As a fourth step in the auditing process, the auditor should verify attainment of follow-up 
measures with a secondary source, especially if the program uses a survey methodology. Compared 
to data matching, surveys are more likely to elicit socially desirable responses. For example, 
students may inaccurately claim to have obtained a job or passed a State recognized high school 
equivalency test because they may believe that attaining these outcomes is expected of them. The 
auditor should (1) contact a sample of employers to verify that the student is or was employed, (2) 
review the high school equivalency test data to verify the claims of those students who claim to pass 
the test, and (3) check enrollment at community colleges to see whether students who claim to enter 
postsecondary programs are actually enrolled. 


DATA SYSTEMS AND NRS REPORTING 


NRS data collection produces a rich source of information about adult education students and 
their outcomes. States and local programs can use these data for program accountability, to identify 
effective programs and instruction, and to foster program improvement. Sections 212(c), 231(e) (2), 
and 212(a) of the Workforce Investment Act (WIA) explicitly identify these purposes in stating the 
reasons and uses for the program accountability system. States must report their performance levels 
on the core measures to ED and use the measures to assess the effectiveness of local programs and to 
promote continuous program improvement. 


This section provides general guidance on establishing a statewide student reporting system 
that allows States to meet NRS requirements. The guidance includes a brief summary of the software 
needs and requirements, a description of the information that must be entered into the student record 
system and the types of outputs or reports that States and local programs should be able to produce. 
Concluding this chapter are tables for reporting NRS data at the Federal level. 


General Software and Architecture Requirements 


To meet NRS requirements, each local program must use an automated, individual student 
record system to enter NRS data. The software for this system must have a relational database 
structure, whereby information on individual students can be related to other variables in the 
database and data can be aggregated and analyzed for specific subgroups. The software also must be 
capable of aggregating data to produce the required Federal reporting tables, or the data must be able 
to be imported into other software that produces the Federal tables. 
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The NRS does not require any specific software product or system beyond these 
requirements. States should carefully consider not only NRS reporting requirements but also their 
reporting needs and the needs and capabilities of local programs when selecting software. Training 
and technical support issues related to software also should be factors when deciding what software 
to use, as should the overall cost of developing and maintaining the system. 


States also should consider the system’s architecture or general structure. There are generally 
two choices: onsite systems and Web-based systems. An onsite system provides separate copies of 
the software individually to each local program. Programs enter the data for their sites into a 
computer located onsite, and all functions are available locally, including reports. The program 
sends its data to a central State computer for reporting. A Web-based system provides access to a 
single, centrally maintained system via the Internet. The system can be used by anyone with an 
Internet connection, browser, and possibly a small piece of software known as an applet. Centralized 
approaches make changes and enhancements to the software easy to implement and eliminates local 
reporting because data are directly entered into a central computer. Exhibit 4.1 offers guidance on 
selecting software. 


Data Structure and Inputs 


The software system should allow local programs to enter and retrieve their own data for 
individual students. To be most useful, the data should be organized by site and class. Exhibit 4.2 
shows the recommended data structure for NRS reporting and analysis. This structure allows 
programs to examine student outcomes by individual class, site, and for the program overall and thus 
provides the greatest ability to examine the relationship among instruction and other program 
components and student outcomes. 


The State and local systems must include at a minimum, the NRS core measures and their 
applicable coding categories and should include basic functions to allow the inputting and reporting 
of these data. Exhibit 4.3 summarizes the basic data elements and functions needed for the NRS. 
The NRS guide, Developing an NRS Data System (available at http://www.nrsweb.org) contains 
more information about developing a data system for the NRS. 


Basic Data System Functions 


In addition to the core measures, States that use the secondary measures should include these 
measures and categories along with any other measures the State needs for its own uses. States and 
local programs also may add coding categories for any core and secondary measures as long as the 
NRS categories can still be reported. For example, States may use additional functioning levels or 
categories for ethnicity. To use NRS data to evaluate program performance and promote program 
improvement, the system also must include other measures, such as information about classes, 
instructors, and program staff. 
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Exhibit 4.1 
Guidance for Selecting Student Record Software To Meet NRS 
Requirements 


Issues in Choosing Software 
Consider the choices for student record software: 


The overall design of the software. 

The training and support offered by the software’s vendor. 

The methods used to enter data into the software. 

The various ways that the software allows the program to use data, including reporting, data analysis, and program planning functions. 


System Design 
Software issues: 
oO What is the cost of the software? 
0 Does the software rely on any other software packages in order to function (e.g., Microsoft Access)? Do local programs have this software? 
o What operating system environment is most appropriate for the software? Is this the system that local programs use? 
Architecture issues: 
0 Does the State want a stand-alone system where every location runs a separate copy of the software and all functions are available locally? 
o Does the State want a Web-based system where users access the system through the Internet with a Web browser? 
Hardware issues: 
0 Dolocal programs and sites have computers that are powerful enough and have enough memory to run the software? 
0 _|s the software available for IBM-compatible or Macintosh computers or both? Are the datasets interchangeable in a mixed environment setting? 


Usability Issues 

Is the software user-friendly or intuitive? 

Do potential users appreciate the appearance of the software? 

Can the software be customized to meet the program’s needs? 

Does the software include the specific measures, coding categories, and data elements needed by the program? 

Can the software be used for multiple years—are its archives accessible from year-to-year, or are only the current year’s records available? 

Can the software be used in a network environment? 

Does the software allow security, such as by limiting access or functionality to specific types of users (e.g., password protection, ability to hide sensitive 
data elements)? 


Training and Support 

Does the vendor offer training and/or support? What mechanisms of training and support are available? 

What is the cost of training or support? 

Does the software have documentation, such as a user's manual? Is the manual helpful and easy to understand? Does it provide useful information? 
Are there planned upgrades for the software? Are software upgrades made available free of charge, and are users notified when they become available? 


Data Input 
The ease of entering data into the software also can be an effective way of differentiating among software packages. 


Are data keyed in manually or can it be scanned into the system? 

0 Is any extra hardware or software necessary for scanning? Can these be leased, or must they be purchased? Do all workstations operating the 
system need these add-ons? 

o What scanning mechanism does the software system support (e.g., scantron/bubble forms, text scanning). 

What does the software use for a student ID number (e.g., Social Security numbers or program-defined numbers)? Can this number be changed if 
necessary? 

Does the software support multisite data entry at the individual program level? Can site-level data be aggregated to the program level? 

Can data be imported from other software packages (e.g., spreadsheets or other databases)? What formats are required by the software for imports? 


Using the Data 
Consider how you plan to use data (i.e., for accountability, program improvement, or program evaluation) and whether the software addresses these needs. 


Does the software come with built-in reports appropriate for the program's uses? 

How difficult is it to create reports or modify existing reports as needs arise? |s additional software needed to create new reports? 

Does the software allow users to search the database for specific records or conduct queries to locate different classes of records? 

What are the analytic capabilities of the software? 

Does the software allow users to conduct analyses at the individual student level? 

Can data be exported to other software packages (e.g., spreadsheets or other databases)? What format does the software use for exports? 
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Exhibit 4.2 
Recommended Data Structure for NRS Reporting and Analysis 


State Adult Education Agency 


Adult Education 
Program A 


Adult Education 
Program Z 


Beginning High 
ESL Class ASE Class Ese Advanced a oe Low ASE Beginning 
y ESL Class y Class ABE Class 
Class Class 


Reporting Capabilities 


Equally important to the system’s data structure and inputs is the system’s capability to output or 
report information. For Federal reporting, the NRS requires that each State annually submit aggregated 
summary tables of descriptive and performance data on the core measures. Each local program’s software 
must have the capability to create these reports and submit an aggregated report to the State, or local programs 
must be able to submit their individual student data to the State for aggregation. 


Although Federal reporting requirements are relatively simple, WIA requires States to use NRS 
data for more extensive purposes. For WIA, States need to evaluate each local program’s performance on 
the outcome measures and address the needs of specific subpopulations, such as low-income students or 
adults in family literacy programs. To obtain this information, the software system must have the 
capability to report by individual program and by student population. 


Even more detailed reporting is needed to use NRS data to address program improvement needs. 
Among the most powerful uses of NRS data is the capability to understand the program and instructional 
factors related to successful student outcomes. To study these issues, States and local programs need the 
ability to examine data by site, class, and student characteristics and to relate outcomes to such variables as 
contact hours, teacher characteristics, and curriculum. While most software systems commonly include such 
data elements, the reporting of this information in a form amenable to program performance evaluation can be 
problematic unless this capability is initially built into the system. 
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Exhibit 4.3 


Basic Data Elements and Functions for the NRS 


Basic Data Elements 

STUDENT INFORMATION 

= Name 

Address 

Phone 

E-mail 

Date of birth 

Gender 

Ethnicity 

Functioning levels 

Test scores and dates 
Program type: 

e ABE 

e ASE 

e ESL 

Highest Degree or Level of Schoo! Completed 
Environment: 

e Family literacy 

e Workplace literacy 

e Homeless 

e Work-based project learner 
e Correctional 

Secondary status measures: 
Low income 

Displaced homemaker 
Single parent 

Displaced worker 
Learning disabled 
Enrollment date 

Separation date 

Attendance hours/dates (weekly/monthly) 


Disability information 
Employment status 
Public assistance 
Community type: 

e Rural 

e Urban 


PROGRAMISITE FUNCTIONS 
= Add program 

= Add site 

= Add class 


= Move sites/classes 


= Class attendance 
STUDENT FUNCTIONS 
Intake 
Enrollment 
Attendance 
Assessment 
Leveling 
Separation 
STAFF FUNCTIONS 
= Staff profile 
= Contact hours 
REPORTING FUNCTIONS 
= NRS tables 
SYSTEM MANAGEMENT FUNCTIONS 
= Assessment/leveling information 
= Cohort designations for follow-up measures 


OUTCOME INFORMATION 
= Core achievements: 
e Entered employment 
e Retained employment 
e Secondary Credential 
e Placed in postsecondary education 
Secondary achievements: 
e Achieved work-based project learner goal 
Left public assistance 
Achieved citizenship goals 
Increased involvement in child’s education 
Increased involvement in child’s literacy activities 
Voted or registered to vote 
Increased involvement in community affairs 
e Student goals for attending 
STAFF INFORMATION 
= Function 
e Teacher 
e Counselor 
e Paraprofessional 
e Local administrator 
e State-level administrator 
= Status 
e Full time 
e Part time 
e Volunteer 
= Teacher Years of Experience in Adult Education 
e Less than one year 
e One to three years 
e More than three years 
= Teacher Certification 
e No certification 
e Adult Education Certification 
e K-12 Certification 
e Special Education Certification 
e TESOL Certification 


DESCRIPTION 
Set up information for program 
Set up information for site associated with program 
Set up information for class associated with site 
Ability to move one or more classes to a different site or sites to a different 
program (merge) 
Enter attendance information for all students in class 


Enter demographics, needs, goals, etc., at intake 
Enroll/drop student in class 

Maintain attendance information for students 
Enter student test scores 

Student level based on test scores (automatic) 
Enter separation information 


Maintain information about staff members 
Enter actual contact hours by week or month 


Generate NRS tables 


Maintain information about test scores and levels 
Maintain information about follow-up cohorts and outcomes 
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Exhibit 4.4 presents examples of the types of tables the software should be capable of 
producing. The first three tables in the exhibit show educational advancement by incoming 
educational level, program area, and class. State and local administrators can use these tables to 
assess program performance standards on this measure and to examine which types of students in 
which classes advance at higher rates. The fourth table in the exhibit offers an example of the type of 
report needed to examine individual class data. In this example, student performance is compared 
with full- and part-time instructors and whether the instructor has had staff development. The NRS 
guide, Using Data for Program Management and Improvement (available at 
http://www.nrsweb.org) contains more information on the many uses of data. 


Federal Reporting Tables 


NRS data are to be reported annually to DAEL by each State in aggregate form. DAEL has 
developed reporting tables for this purpose. These tables—included in appendix C—have been 
revised by DAEL and reviewed and cleared by the Federal Government. Instructions on completing 
each table are included with that table. 


Several optional reporting tables are provided to allow for separate reporting about special 
populations on the core indicators. For example, tables for workplace and family literacy 
participants provide a picture of how the participants performed on core and secondary measures. 
States are encouraged to examine the performance of other target subpopulations separately. 


Employment measures reported in Tables 5 and 5a follow a multiple year reporting 
procedure. A time lag in the availability of employment data from the UI data base used for 
data matching requires reporting of students who attended in different program years for entered 
and retained employment measures. 


e Reporting Entered Employment. Data for students exiting in the Second, Third, and 
Fourth Quarters of a program year and the First Quarter of the next program year will all be 
reported under the next program year. For example, the data for a student who exits in 
October of 2007 (Second Quarter PY 2007) will be reported in the PY 2008 report (due 
December 2009). 


e Reporting Retained Employment. Data for students exiting in the Fourth Quarter of the 
previous program year and the First, Second, and Third Quarters of the current program year 
will all be reported under the next program year. For example, the data for a student who 
exits in April of 2006 (Fourth Quarter PY 2005) will be reported in the PY 2007 report (due 
December 2008). 


See NRSWeb.org for further information. 


In addition to data tables, DAEL requires States to submit a narrative report and a financial 
report detailing expenditures. States receiving funds under the EL Civics program must complete a 
separate financial report for the funds. Forms and instructions for both of these financial reports also 
are in the following section. 


States are required to submit the reporting tables 6 months after the end of the program year 
(December 31), using DAEL’s electronic submission Web site. 
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NRS Reporting for Students in Distance Education 


States will report all required NRS data elements on distance education students in all NRS 
tables, according to current requirements. States electing to develop proxy contact hours for students 
in distance education will report both proxy and actual contact hours in Table 4. 


States must report data on students in distance education separately in Table 4c, identical to 
NRS Table 4, and in Table 5a, identical to Table 5. Only students in distance education are to be 
reported in these new tables table and all contact hours (proxy and actual) are to be reported in Table 
Ac. 
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Exhibit 4.4 
Sample Tables for Examining Program Improvement 
and Program Effectiveness 


Educational Advancement Information 


Average Contact 
Hours Per Student 
Before Advancement 


All Levels 


Number Recommended Percentage of Students 


eS re. for Advancement Advancing by Level 


Number Enrolled Number Recommended for 
(all levels) Advancement 


Educational Advancement by Class 


Average Contact 
Class Percent Advancing Pretest Score Range Hours Per Student 
Before Advancement 
Beginning Literacy Class 1 162-204 
Beginning ABE Class 1 


All Classes 
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Exhibit 4.4 (Continued) 
Sample Tables for Examining Program Improvement 
and Program Effectiveness 


Instructor (Class) Low Intermediate Level 
Name Pretest Posttest Gain 


Angeles, January 
Arrendondo, Myra 


| James, Brad 


es ee ee ee 
ee ee ee ee 
212 209 3 15 | 
Martinez, Juan 
Mulligan, Ivor 
Simone, Michael 
i 4 4 


Sauti, Christina 
Soden, David 
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Exhibit 4.4 (Continued) 
Sample Tables for Examining Program Improvement 
and Program Effectiveness 


Instructor (Class) Low Intermediate Level 
Name Pretest Posttest Gain Yes No Yes No 
Thompson, Terty walt, ir oie, We ao I 
A | 
| BenMartinez(Class#3) | ay aes | a ee a ee a | 


a a (| ea) 
7 a a: | Y a 
7 | 4 |  « | | ae 
2 | 8 | pj. | 
219 | HT PO 
211 a aaa 
222 Sa aaa 
215 P= 2 
211 a 
220 Ee == 4) 
213 a ea 
214 [=| -§—___] 
Average 


213.8 
Aladjem, Daniel 219 


RS] RS] DO] DOT PO 
=] =] =] a] Sa 
Be] WN] —] >] co 


Ferrara, Steve 


Flores, Bernardo 212 211 -{ 75 
215 = 78 a a a | | 
Gonzales, Jesus 213 211 -2 75 
211 = i a | | 
210 1 81 es A Si PSS 
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Exhibit 4.4 (Continued) 
Sample Tables for Examining Program Improvement 
and Program Effectiveness 


Low Intermediate Level 


Instructor (Class) 


Pretest 

Snow, Stephanie 214 

a 9 90 
211 2 75 
220 +7 90 
220 +6 93 
Jiang, Tao 211 217 +6 84 
216 +6 87 
220 +8 84 
217 #2 83 
216 = 78 
2 7 78 
223 +5 90 
276 | #38 aa 
Carpenter, Daniel 

219 4 83 
Mesmer, Eric 214 214 — 81 
[a 
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Instructor (Class) Low Intermediate Level 
Name Pretest Posttest Gain 
+d 
+d 
+5 
5 
132 
enniferLenis(Gassh) | 
mM = 78 
213 2 87 
215 87 
Johnson, Tony 


Arlinda Morris (Class #8) ee 


Millstone, Ken 


Woodford, Alix 
Average 
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SAMPLE LOCAL FOLLOW-UP SURVEY FOR CORE MEASURES 


A. ENROLLMENT 


Hello. My name is . | work for . We're calling people who have recently attended classes at 
our adult education program to find out what happens to them after they leave us. We want to know how you liked the classes 
you took and how adult education classes have affected you, your family, and your job. 


It should take no longer than 10 minutes to answer my questions. Do you have time now for me to ask these questions? 
(Reassure the respondent that any information given to us will be strictly confidential.) 


First, I'd like to make sure | have the correct information about the class you took. 


A-1. | understand that you were in (TEACHER’S NAME)’s class at (LOCATION). Is that correct? 


Q Yes 
Q No [Obtain correct information] 


A-2. Did you attend teacher’s class until it ended or did you leave before it ended? 


Q Completed [Proceed to Question B-1] 
Q Left before it ended [Proceed to Question A-3] 


A-3. During what month did you stop attending the class or program? 
Month 
B. OTHER EDUCATION AND TRAINING 
B-1. Since the end of your class or program, have you enrolled in any other educational or training programs? 
Q Yes 


Q No [Proceed to Question C-1] 


B-2. Where are you enrolled? 
Q = Other (Specify) 


B-3. In what type of class or classes are you now enrolled? [Do not read choices to respondent. Check all that 
apply.] 


English Language Skills 

High School 

Vocational/Job Training 
Community College/College Level 
Citizenship 
Family literacy 
Other (Specify) 
Dk/Refused 


OUoCOCOCOCD 
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C. SECONDARY CREDENTIAL 


C-1. Did you receive any diplomas, certificates, or degrees at the end of your class or since you left (TEACHER’S) 
class? 


Q Yes 
Q No [Proceed to Question D-1] 
Q DkK/Refused [Proceed to Question D-1] 


C-2. What type of diplomalcertificateldegree did you receive? [Do not read choices to respondent. Check all that 
apply.] 
Q High School Diploma 
Q High School Equivalency 
Q Certificate of Competence 
Q Associate’s Degree 
Q Bachelor's Degree 
Q Other. 
Q Dk/Refused 
D-1, When you first enrolled in the class or program were you: [Read choices.] 


Q Employed at a paying job [Proceed to Question D-4] 

Q Not employed at a paying job and looking for a job [Proceed to Question D-2] 
Q Notemployed and not looking for a job [Proceed to E-1] 

Q  DK/Refused [End interview] 


D-2. While you were taking (TEACHER’S)’s class, did you get a paying job? 


Q Yes 
If yes: What was the name of your employer? [Proceed to Question D-4] 
Q No 


D-3. Since you stopped taking the class, have you gotten a paying job? 


Q Yes 

If yes: What is the name of your employer? 
When did you first get a job after leaving the program? 

Q No/Proceed to Question E-1] 


D-4. Do you still have that job or do you now have a different job? 


Q = Still have same job 

Q Have different job 

What is the name of your current employer? 
Q Lost job, unemployed 

Q  DkK/Refused 
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CLOSING 


Thank you very much for taking the time to answer my questions. Your answers will be very 


helpful. The information you gave me will be used to help make adult education programs better and 
more useful to people like you who have attended or would like to attend such a program. 


E-1. Is there anything that | didn’t ask about that you'd like to say? 
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Interviewer: 


CONTACT LOG 


Date & Time 


Name 


Contact (who, nature of conversation, 
any messages left, etc.) 


Status 
(interview completed, 
scheduled recall) 
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SAMPLE FOLLOW-UP SURVEY FOR CORE, SECONDARY, 
AND OTHER MEASURES 


Hello. My name is . | work for . We're contacting people who have recently attended our 
classes at our adult education programs to find out what happens to them after they leave us. We also want to know how you 
liked the classes you attended and how adult education classes have affected you, your family, and your job. 


It should take no longer than 15 minutes to answer my questions. Do you have time now for me to ask these questions? 
(Reassure the respondent that any information given will be strictly confidential.) 


ATTENDANCEIOBJECTIVES 


A-1. | understand that you were in (TEACHER’S NAME)’s class at (LOCATION). Is that correct? 


Yes 
No [Obtain correct information] 


A-2. During what month and year did you enroll in this program? 
Month Year 
A-3. Did you attend the class/program until it ended? 


Yes [Proceed to question B-1] 
No [Proceed to question A-4] 


A-4, During what month did you stop attending the class or program? 
Month 
A-5. What was the main reason you stopped attending the class or program? [Do not read choices to respondent. 
Check category that is most closely related to response.] 
Achieved reason for enrollment Instructor was not good 
Completed class Program didn't satisfy personal goals 
IIlness/Incapacity Not satisfied with program 
Lack of child care Moved 
Lack of transportation Entered employment 
Family problems Entered other education or training program 
Time or location of services not feasible Other (Specify: ) 
Lack of interest DK/Refused 
Instruction not helpful 


SECONDARY CREDENTIAL 


B-1. Did you receive any diplomas, certificates, or degrees since you took this class? 


Yes [Proceed to question B-2] 
No [Proceed to question C-1] 
DK/Refused [Proceed to question C-1] 
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B-2. What type of diplomalcertificate/degree did you receive? [Do not read choices to respondent. Check all that 
apply.] 

High School Diploma 

High School Equivalency 

Certificate of Competence 

Associate's Degree 

Bachelor's Degree 


Other. 
DK/Refused 
C-1. Since you stopped attending the class or program, have you enrolled in any other educational or training 
programs? 
Yes 


No [Proceed to question D-1] 


C-2. Where are you enrolled? 
Q = Other (Specify) 


C-3. In what type of class or classes are you now enrolled? [Do not read choices. Check all that apply.] 


English Language Skills 

High School 

Vocational/Job Training 
Community College/College Level 


Citizenship 
Family Literacy 
Other (Specify) 
Dk/Refused 
D-1. While you were enrolled in the class or program, were you receiving any type of public assistance, such as 


food stamps or welfare benefits? 


Yes 
No [Proceed to question D-3] 
DK/Refused [Proceed to question D-3] 


D-2. Are you currently receiving this type of public assistance? 
Yes 
No 
DK/Refused 
D-3. When you first enrolled in the class or program, were you: [Read choices.] 


Employed at a paying job [Proceed to question D-6] 

Not employed at a paying job and looking for a job [Proceed to question D-4] 
Not employed and not looking for a job [Proceed to question E-1] 
DK/Refused [Proceed to question E-1] 
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D-4. While you were taking this class, did you get a paying job? 


Yes 
If yes: What was the name of your employer? [Proceed to Question D-6] 


No [Proceed to question D-5] 


D-5. Since you stopped taking this class, have you gotten a paying job? 


Yes 
If yes: What is the name of your employer? [Proceed to Question D-6] 


When did you first get a job after leaving the program? 


No [Proceed to E-1] 


D-6. Do you still have the same job, have a different job, or have no current job? 


Still have the same job 

Have a different job 

What is the name of your current employer? 
Have no job, unemployed 


DK/Refused 
CommUNITY IMPACT 
E-1. Compared to before you attended the class, have you increased your attendance or activities in any of the 


following: [Read choices. Check all that apply.] 


Neighborhood meetings 

Meetings of political groups 

Volunteer work or meetings for community organizations 

(List: ) 
Do not go to meetings or volunteer 

DK/Refused 


E-2. Did you register to vote or vote for the first time since you attended the class? 


Yes 
No 
DK/Refused 


FAMILY 


F-1. Do you live with children who are 12 years old or younger? 


Yes 
No [Proceed to question G-1] 
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F-2. Since you attended the class, how much do you read with your children compared to before you attended the 
class? Do you: 


Read with children about the same as before 
Read with children more than before 

Read with children less than before 

Do not read with children at all 


DK/Refused 
F-3. How often do you visit the library with your child/children now compared to before attending the program? Do 
you: 
Q Go more often 
Q Go the same amount 
Q~ Goless often 
Q Not go atall 
Q  DK/Refusal 
F-4, Is/are the child/children in your home attending school? 
Q Yes [Proceed to question F-5] 
Q No [Proceed to question G-1] 


F-5. Compared to before you attended the class, how much time do you spend helping the school-aged children in 
your home with homework? Do you: 


Help about the same 
Help more than before 
Help less than before 
Not help at all 
DK/Refused 


F-6. Compared to before you attended the class, how many of your children’s school activities, including 
parentiteacher conferences and school assemblies, have you gone to? 


Attend about the same 
Attend more activities 
Attend fewer activities 
Do not attend activities 
DK/Refused 


SATISFACTION WITH PROGRAM 


G-1. What is your general opinion of the quality of the class you attended? Is it unacceptable, not very good, 
satisfactory, or excellent? 


Unacceptable 
Not very good 
Good 
Excellent 
DK/Refused 


G-2. Did (TEACHER’S NAME)’s class meet the expectations you had for it before you enrolled in it? 


Yes 
No 
DK/Refused 
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G-3. Are you not at all likely, somewhat likely, or extremely likely to attend another class or program offered by 
(PROGRAMICLASS ORGANIZER)? 


Not at all likely 
Somewhat likely 
Extremely likely 
DK/Refused 


G-4. What did you like about this class or program? [List all responses.] 


G-5. What did you not like about this class or program? [List all responses.] 


CLOSING 


Thank you very much for taking the time to answer my questions. Your answers are very helpful. The information you gave me 
will be used to help make adult education programs better and more useful to people like you who have attended or would like to 
attend such a program. 


H-1. Is there anything that | didn’t ask about that you'd like to say? 
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MODEL PROCEDURES FOR CONDUCTING THE 
LOCAL FOLLOW-UP SURVEY 


This section describes model procedures for conducting a survey designed to collect the NRS 
follow-up measures. The model is offered as guidance to States in designing and conducting the 
follow-up survey. These procedures are not required, however, and States may develop its own 
procedures for conducting the survey, as long as the survey meets the NRS requirements described in 
this document. 


The crucial activities to conducting a survey that produces valid data are to: 


iF 


3. 


Draw a sample of students that reflects the students who attend your program with one or 
more of the four core outcomes. 


Reach the students sampled and obtain the information from a large majority of them so 
as not to invalidate the sample. 


Train interviewers so that all interviewers ask the survey questions correctly and reliably. 


Selecting the Sample 


The procedures below present a method for randomly selecting a sample of 300 students who 
left the program. 


1. 


Generate a list from your database of names, with contact information, of students who 
have left the program, and organize the list based on the four core follow-up measures. 
Use students’ outcome cohorts to develop the groups. You should also have the exit 
quarter for students for employment outcomes. You may use separate lists for each of the 
four follow-up measures or a single list with all students. 


Go through the list to identify any individuals who do not have any contact information. 
Cross these names off your list. 


We want to compute a sampling fraction that will give us a 300 student sample. The 
sampling fraction is computed by dividing the total number of students to be sampled by 
the desired sample size. If we have 600 students, the sampling fraction is 600/300 = 2. If 
the sampling fraction is not a whole number, it should be rounded to the nearest whole 
number. 


From your list, count down the number of students determined by the sampling fraction 
and include that student in the sample and continue this way throughout the entire list. 
For example, if the sampling fraction is 2, then include the second student on the list and 
every other student thereafter. When you are finished, you will have a sample of 300 
students. This is your primary sample. 


Create a backup sample of 50 percent more students than your primary sample. 
Randomly select the backup sample in the same way as the original sample. Compute the 
sampling fraction by dividing the number of students you need by the number of students 
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remaining on the list. Use this number to select every n'" student from the list. Make 
sure you have a backup sample sufficient for each of the four core outcome measures. 
The backup sample is used to replace students from the primary sample who cannot be 
reached after four attempts (see below). If there are fewer students remaining on your list 
than you need for the backup sample after you have selected your primary sample, then 
all of these students will be included in the backup sample. 


Survey Procedures 


Once you have your sample, you can begin contacting students and administering the survey. 
Contact each person on your primary sample list. If you cannot reach a person despite your best 
efforts, then replace that student with a student from the backup sample. 


As you conduct the survey, it is very important to the integrity of the data collected to know 
how many people in the sample were not reached, how many refused to participate, and what the 
reasons for refusal were. For this reason, maintain a contact log during the survey. Entries in the log 
should contain the date and time of each contact, the name of the interviewer, and information about 
each contact, including: the name of the respondent, whether the person was reached, messages left, 
whether the interview occurred, and explanations for why it did not. The logs should be checked 
daily to identify respondents who need to be recontacted. The log should also be checked against the 
list of learners in the sample to make sure all members of the sample are being contacted. 
Interviewers should promptly make a log entry for each contact they make, whether or not the adult 
learner was reached. This Appendix includes a sample contact log. 


The validity of the survey depends on reaching all or at least a majority of the students in the 
sample. There will be many difficulties, however, in reaching all of the students in the sample. The 
following section describes some of the most common difficulties in reaching people for a survey 
and offers guidance on how to resolve these problems. 


Problems Reaching Learners 


In most data collection activities, there are predictable kinds of problems that may be 
encountered. Interviewers may be unable to reach the correct person, and the learner may not want 
to speak to the interviewer, or he or she may have a protective family. Additionally, learners may not 
want to answer some or all survey items; they may be hostile, confused, or just harried. Furthermore, 
callers may be required to answer questions that they are not equipped to answer. 


Interviewers should have a resource person available who can assist with difficult interviews 
or respondents and complicated questions. This person should have thorough familiarity with the 
NRS and the procedures used to conduct interviews. He or she should monitor interviewer contact 
logs, provide general oversight during the interviewing process, and could also be responsible for the 
training. 


Accommodation for other languages. Because the sample may include ESL students and 
other non-native English speakers, interviewers are likely to encounter a language barrier in the 
course of data collection. Every effort must be taken to collect information from all non-English 
speakers included in the sample. Accomplishing this may require the program to translate the survey 
and use interviewers who are fluent in the languages that may be encountered during the interviews. 
The NRS has Spanish and Vietnamese versions of the model survey that are available on request. 


NRS Implementation Guidelines A-13 


Appendix A: Sample Surveys 


When the student cannot be reached immediately. A gatekeeper is a person or situation 
that stands between you and the person with whom you need to talk. Common gatekeepers are 
family members and voicemail or answering machines. 


Reaching a family member or other person 
v Leave a message. The message should be as follows: 
(| Interviewer’s name and where interviewer is from (name of program). 


Contacting in reference to the adult education program the person attended. 


Interviewer will try contacting learner another time. 
v Ask a few questions: 


When is the learner expected back? 


What and when is the best way to reach him/her? 
¥ Wait for no more than 2 days between attempts to contact the learner. 


v If multiple messages (more than 3 or 4) have been left, but the learner has not been 
contacted, then the learner should be officially listed as a nonrespondent on the 
contact log sheet and replaced with a learner from the backup sample. 


(| Reaching voicemail or an answering machine 


v Leave a message. The message should be as follows: 
Interviewer name and where interviewer is from (name of program). 


Calling in reference to the adult education program the person attended. 


Interviewer will call back at another time. 
v¥ Wait no more than 2 days between callbacks. 


v If multiple messages (more than 3 or 4) have been left, but the learner has not been 
contacted, then the learner should be officially listed as a no-respondent on the 
contact log sheet and replaced with a learner from the backup sample. 


(| Reaching a non-working number or a number that just rings 


¥ Non-working number should be noted on the calling log sheet as not working. 


v If the number just rings, then the day and time the interviewer called should be noted 
on the log sheet and the learner should be called at a different time. If multiple calls 
(more than 3 or 4) are made at different times of the day and there is still no answer, 
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then the learner should be officially listed as a nonrespondent on the contact log sheet 
and replaced with a learner from the backup sample. 


Dealing with refusals. The goal of interviews is to obtain information from all the people 
contacted. However, some interviewees may be initially reluctant to participate in the survey. The 
interviewer should try to “convert” refusals whenever possible; interviewers should, however, never 
become belligerent or upset or insist that a person complete the survey. 


The best way to handle a refusal is for the interviewer to present himself or herself as 
confident and proud of the work that he or she is doing. The interviewer should indicate that this 
survey is an important way of providing information to the State Department of Education and the 
adult education program, and decisions about adult education will be made based on this information. 


There are several points in the interview when interviewers may encounter refusals or 
reluctance. The following examples provide ways to handle this. 


Initial refusal. When learners are first reached, they may not be prepared to speak with the 
interviewer. They may be very busy. If this is the case: 


e Ask about the timing: I’m sorry we reached you at a bad time. When might be a more 
convenient time to reach you? Possible solutions include offering to contact them a week 
later, a month later, etc., as long as this is recorded so that the follow-up call is made. 


e When the learner has been reached but absolutely refuses to participate, a complete 
description should be recorded on the contact log and given to the resource person for 
further attempts. 


Confusion-based refusal. Adult learners who are contacted may be confused or wary about 
how the information collected in the interview will be used. For this reason, they may refuse to take 
part in the interview. 


e If the learner wants to know why the survey is being conducted, the interviewer should 
explain the purpose of the study, emphasizing that the information collected has 
important implications for the national adult education program and for the program she 
or he attended. 


e If the learner wants to know how his or her information will be used, the interviewer 
should assure the learner that the data will be compiled to find out how well adult 
education programs are performing throughout the country and to improve program 
services. Furthermore, all of the answers that the learners give will be kept confidential, 
and no names or other identifying information will be associated with their answers. 
Learners should also be assured that they were chosen randomly from the pool of adult 
learners in the State. 


Time- or burden-based refusal. This type of refusal can occur early in the interview or at a 
later point. Interviewees may be pressed for time and may try to terminate the interview. If this is 
the case: 
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e The interviewer should point out that the survey will only take 10—15 minutes, 
acknowledge that the learner’s time is really important, and tell them that their responses 
to the survey questions will be really helpful: I understand that your time is important. 
We really appreciate your input on this issue. It is important to get the perspective of 
adult education students. 


e The interviewer should tell learners about the sampling process: Of the [number] 
students that attended the adult education program, you have been selected as one of only 
[number] to represent the program. Your help is important to us. 


If the respondent is still reluctant, one other strategy may be helpful: 


e The interviewer should try to arrange an alternate time: Might there be a better or more 
convenient time to contact you? 


If none of these strategies are successful, then the interviewer should NOT try to persuade the 
learner further. The learner should be thanked for his or her patience and told that the caller 
appreciates all the demands on their time. The interviewer should then record a complete description 
on the contact log, and the student should be replaced with a learner from the backup sample. 


Training 


Staff members who will be conducting the interviews should be trained to ensure the integrity 
of the data collected. To collect valid and reliable data, interviewers must be thoroughly familiar 
with both the process of interviewing and the materials to be used for collecting data. The actual 
training can be characterized as having two components: the process of conducting interviews and 
the purpose and structure of the NRS. This section provides suggestions on appropriate training 
activities. 


Focus of Training 


Regardless of the survey, any errors, biases, or inconsistencies on the part of the interviewer 
result in some degree of survey error. It should be a goal to minimize this error. Trained 
interviewers are much more likely to accomplish this goal. The desired result is high quality data, so 
that data are comparable from one interview to another, as well as from one state to another. The 
following guidelines should help minimize survey error, and should thus be conveyed to the 
interviewers during their training. 


1. The interviewing process should be standardized. To ensure that this occurs, interviewers 
must read the questions exactly as written and follow the instructions on the survey 
instrument. 


2. Interviewers should avoid biasing answers by not showing criticism, surprise, approval, 
disapproval, and/or annoyance at a response; recording answers promptly and accurately; 
and probing for clarification when necessary. 


3. Interviews should be completed in the time promised to the respondents. The interview 
is designed to take about 10 minutes. 
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4. Interviewers must be familiar with the material, including the meaning of individual 
questions and the definitions of words and phrases contained in the survey instrument. 


5. Administrative issues should be attended to as soon as possible, including making a 
record of EVERY call made, even if the interviewers reached a wrong number, if nobody 
answered, or if a message was left. 


6. Interviewers should have a thorough understanding of the purpose and structure of the 
NRS and the pilot, as explained in this manual. 


Conducting the Training 


Training interviewers can take many forms, including workshops and meetings. There are, 
however, a few techniques which will make the training more meaningful, and thus make the data 
collected more useful and comparable between states. Among these techniques are: 


e Going over the protocol question-by-question. This will give interviewers a familiarity 
with the questions and answers they are likely to get during their telephone calls. It will also 
allow them to become comfortable with the decisions that must be made as the interview 
begins. 


e Conducting mock interviews with adult education office staff or teachers. This simulates 
real world conditions, giving interviewers valuable practice on how to conduct interviews. It 
also allows adult education office staff to identify issues that were not made clear earlier in 
the training process and to identify problems with the data collection procedures in place. 


Conducting a mini-pilot test with students not included in the official NRS sample in the state. 
This activity will identify previously unconsidered issues and provide the most realistic training for 
the interviewers. It is an excellent last step prior to officially collecting 
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<A 
NRS 


National Reporting System 
for Adult Education 


A Project of the U.S. Department of Education 


NRS STATE DATA QUALITY STANDARDS CHECKLIST 


Instructions for Completing the Checklist 


States use this checklist to rate their implementation of the data quality standards in their 
NRS data collection procedures. States also describe details of their data collection policies and 
procedures for some standards. States must also include with the checklist, a plan for data quality 
improvement. The director of the administrative State agency where the Federal adult education and 
literacy program resides must certify the checklist, and the checklist must be submitted with the 
annual NRS data performance report. 


Reporting on Standards 


The checklist presents the standards for each content area and quality level. States are to 
report whether they have the policy process or procedure described by the standards by indicating 
“yes” if the standard has been met or “no” if it has not been met. Some standards apply only to the 
survey or data matching follow-up methodologies for collecting NRS outcome measures. If the State 
does not use one of the methods, then it should report “not applicable” (NA) for the standard. Please 
note that because your State may meet some of the standards in all categories (e.g., some under the 
acceptable level and some under the superior level), it is important to complete the entire checklist. 


To report that the State has met the standard (i.e., “yes”), the State must meet all of the 
criteria for the standard. For example, for the standard concerning written State assessment policy, 
the policy must include all of the topics listed in the standard. Otherwise the State must indicate “no” 
for the standard and address the problem in the data quality improvement plan. 


Narrative Detail 


Some standards require the State to provide additional information, such as the name of 
assessment used, the State’s follow-up method, or a narrative description with more detail. For 
example, if a State has a system of technical assistance on data quality to local providers, then the 
State must describe the system. All narrative descriptions should be brief but sufficient enough to 
convey the information requested. No more than a few sentences are necessary. 


Data Quality Improvement Plan 


If a State fails to meet superior standards in any area, the State must include a brief data 
quality improvement plan that describes how it will move toward superior quality within the next 
year. The plan must address all standards that the State did not meet, describe what new policies or 
procedures it will put in place to meet the standards, and identify barriers to moving to a higher 
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quality level and the technical assistance needed to implement the plan. DAEL will offer technical 
assistance to States. 


Submission and Certification 


States are to complete the checklist for the program year for the NRS data due on December 
31 of each year. The last page of the checklist is a certification page, where the State director of 
adult education or head of the administrative State agency where the Federal adult education and 
literacy program resides must certify to the accuracy of the information in the checklist. The director 
or agency head must sign this page. All signed reports and data certifications should be sent as a 
scanned email attachment. Mailed paper with signatures will no longer be required. 
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<A 
NRS 


National Reporting System 
for Adult Education 


A Project of the U.S. Department of Education 


NRS STATE DATA QUALITY CHECKLIST 


State: Date: 


Completed by (name and title): 


A. Data Foundation and Structure 
Acceptable Quality 
1. State has written assessment policies that specify: Yes [_] No L_] 


= Standardized assessments to use for accountability that are valid and appropriate for 
adult students. 

= Time periods (in hours or weeks) for when to pre- and posttest. 

" Score ranges tied to educational functioning levels (EFL) for placement and for 
reporting gains for accountability. 

= Appropriate guidance on tests and placement for special populations (e.g., students 
who are unable to be tested due to language or disability). 

= Unacceptable methods of assessment for EFL placement. 

= Appropriate guidance on requirements and conditions for testing distance education 
students reported in the NRS (if applicable). 


1a. List up to three of the most of commonly used assessments permitted for ABE and 
ESL. 


ABE Assessments: 


ESL Assessments: 
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2. State has written policies for follow-up that explain: Yes] No [] 


= Which students are to be followed. 

= How to determine tracking cohorts. 

" Follow-up methodology (survey or data match) for each measure that meets NRS 
requirements. 


2a. Indicate your follow-up methods for each measure. 


Entered employment: Survey [_]  Datamatch[_] Both (explain)_] 


Retained employment: Survey[_] -Datamatch[_] Both (explain)[_] 


Secondary credential: Survey [_] Datamatch[_] Both (explain)_] 


Enter postsecondary: Survey [_] Datamatch[_] Both (explain)[_] 


2b. If you indicated that you use both survey and data match for follow-up methods for one 
or More measures in 2a, please explain. 


3. If state uses survey follow-up method for any measure (check not applicable (NA) and 
skip to the next item if survey is not used): NA 


" Local programs or state can produce a list of students to survey, according to NRS 
requirements. Yes [] No 
= Survey is conducted with a state provided, standard survey instrument. 
Yes [|] No [] 
= State has a regular schedule (e.g., quarterly) for submission of survey data or student 
names from local programs. Yes |] No 


4. If state uses data matching for any measure (check not applicable (NA) and skip to the 
next question if data matching is not used): NA 


" Local or state data system can produce files for matching that include exit dates and 
employment status for each student. Yes [] No [] 
« State has established a procedure for collecting Social Security numbers or other 
unique identifiers, including how to deal with missing numbers. 
Yes [|] No [] 
= State has set a regular schedule for data submission from local programs and for data 
matching with external agencies. Yes |] No 


5. States has provided to all local programs a copy of the assessment policy and an 
explanation of the policy. Yes [] No 


6. State has provided to all local programs the written state policies, procedures and 
requirements for student follow-up and an explanation of the procedures. 
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7s 


Yes [|] No [] 


The state has written definitions for all measures (including demographic measures and 
actual or proxy contact hours, if applicable), defined according to NRS requirements and 
has provided them to all programs. Yes [] No [] 


The State has written policies on the use of proxy hour models to assign the proxy hours 
for distance education learners participation in adult education distance education 
programs. (Check NA and skip to the next item if proxy hours are not used) 

NAL] Yes[_] No [] 


8a. If yes, please identify which model or models were used to assign proxy hours 


Clock Time Model 
Teacher Verification Model [_] 
Learner Mastery Model L] 


Superior Quality 


I 


The state has a comprehensive data dictionary, which defines all measures on state 
student data forms and in the state data system, and has provided it with an explanation to 
all local programs. Yes] No [] 


State has standards or requirements for the percentage of students to be pre- and 
posttested. Yes |] No [] 


2a. If yes, indicate the standards or requirements. 


State has made available to local programs on a continuous basis additional technical 
assistance and resources on assessment, data collection and follow-up procedures (e.g., 
site visits, contact persons, manuals, online resources). Yes [] No [] 


3a. If yes, briefly describe the assistance and how it is provided. 


If state uses survey follow-up method for any measure, the state has taken steps (e.g., 
through data review, discussion with staff or observation) to verify that the survey is 
being conducted according to NRS guidelines (check NA and skip to the next item if 
survey is not used). NA [] Yes [] No [] 


Aa. If yes, briefly describe your verification procedures. 
If state uses survey follow-up method, the state has provided written guidance or 


assistance on how to improve response rates to survey staff (check NA and skip to the 
next item, if survey is not used). NAL] Yes [] No 
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6. If state uses data matching, the state has written procedures on how to conduct data 
matching that comply with NRS guidelines (check NA and skip to the next item if data 
matching is not used). NA [] Yes [] No [] 

7. State has procedures in place that verify whether proxy hours are calculated and assigned 
appropriately (check NA and skip to the next item if proxy hours are not used). 

NA[] Yes[.] No [] 

Exemplary Quality 

1. State has a system for verifying that local programs are following state data policies and 
procedures through program reviews, auditing or a certification process. 

Yes [|] No [] 
la. If yes, briefly describe your verification procedures. 
2. State has reviewed technical reports or research studies of the comparative reliability and 


validity of the NRS assessments it uses to accurately evaluate EFL completion 
performance of its students. Yes [_] No [] 


2a. If yes, briefly describe the technical reports or research studies you reviewed. 


Data Collection and Verification 


Acceptable Quality 


1. 


The state has an electronic management information system (MIS), used by all programs, 
that has individual student records within a relational data base structure. The MIS 
incorporates NRS measures using common definitions and categories. 


Yes [|] No [] 


Database has error checking functions used by state and/or local programs (e.g., that 
identify out-of-range values and missing data). . Yes [] No [] 


State has standardized forms (electronic or paper) for collecting student information (e.g., 
intake, attendance) that include all NRS measures and have correct NRS definitions and 
categories. . Yes [_] No [] 


All programs are required to use state student data forms. Yes] NoL] 


State has provided to local programs guidelines or procedures for recording actual and, 
if applicable, proxy contact hours that conform to NRS requirements. 


Yes [|] No [] 
All or most local programs have staff with clear responsibility for data collection and data 
entry. Yes [|] No [| 
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7. State staff checks data for errors after submission by local programs. 


Yes [|] No [| 
7a. If yes, explain error checking process, including what data are checked and how 
often. 


Superior Quality 

1. Programs and/or state enter data into MIS at least quarterly. Yes [_] No [] 

2. State staff reviews local data at least quarterly for errors, missing data, out-of-range 
values and anomalous data, and to identify program improvements and accomplishments 


and has a system to resolve them. Yes [] No [] 


3. State has timely (e.g., quarterly) follow-up back to local programs to have them correct 
missing and erroneous data. Yes [|] No [] 


4. State has documented procedures for correcting errors and resolving missing data that 
programs use. Yes [|] No [] 


Aa. If yes, briefly explain your data review and error correction system. 


5. State provides additional technical assistance to local programs with poor data, as needed. 


Yes[] NoL[] 
Exemplary Quality 


1. State has a regular system for verifying (through software, onsite auditing, contact with 
local staff) that local programs are following state data collection procedures. 


Yes[.] NoL[] 
la. If yes, briefly describe the methods used for verification, including use of the correct 


assessments and assessment forms, reporting of accurate score ranges for placement and 
for reporting advancement for accountability. 


2. State verifies data have been corrected in state or local database after errors have been 
found. Yes[] NoL[] 


3. State has procedures for regular contact with local staff on data issues to identify 
problems and provide assistance. Yes] NoL[] 


3a. If yes, specify procedures and type of contact. 
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4. 


If state uses survey follow-up method, state tracks survey response rates on at least a 
quarterly basis and takes corrective action if problems are identified (check NA and skip 
if survey is not used). NA [] Yes[] NoL[] 


Data Analysis and Reporting 


Acceptable Quality 


ie 


The state MIS can produce NRS required reports for state, including federal NRS tables 


Yes [] No [] 


NRS tables are calculated accurately to include error checks and prevent double counting. 
Data is reported using minimally acceptable levels of response as specified in the 
reporting tables. Yes] NoL] 


State staff (or designee) checks NRS reports for errors and missing data and obtains 
corrected data from local program reports. Yes] NoL] 


The MIS is capable of reporting disaggregated data by subpopulation (e.g., student age, 
race, sex) and program (e.g., ABE, ESL, ASE, correctional education, distance 
education). Yes [|] No [| 


Superior Quality 


1. 


State staff person familiar with the data, but not directly involved with collection and data 
entry, reviews NRS data reports for errors and accuracy. Yes] NoL[] 


State staff uses data for program management and improvement. 


Yes [] No [ ] 


2a. If yes, provide at least one example of use of data for this purpose in the last year. 


Local programs can access data reports that are useful for program management and 
improvement. Yes [|] No [] 


3a. If yes, briefly describe the usefulness of two reports produced by your system. 


4. Local staff uses data for program management and improvement. 


Yes [|] No [] 


Exemplary Quality 


di. 


State has a system of regular contact with local programs on data analysis issues and 
reporting needs to identify technical assistance needs. Yes] NoL[] 
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la. If yes, specify method and frequency of contact. 


2. State has documented procedures for dealing with analysis problems and deviations. 


Yes [|] No [| 


3. State compares data among programs and with prior years’ data for discrepancies, 
reasonableness and to identify trends in good and bad performance. 


Yes [] No [ ] 


4. State has procedures to verify that local reports accurately reflect data collected (e.g., 
through review of local program documentation, onsite auditing). 


Yes [] No [ ] 


Aa. If yes, describe the report verification process. 


Staff Development 
Acceptable Quality 


1. Local programs and state staff have been provided training on general NRS requirements, 
including assessment policy and procedures, and follow-up policies. 


Yes [|] No [ | 


la. If yes, briefly describe when the most recent training occurred, its duration and about 
what percent of local providers attended. 


2. Local staff has received training on data collection procedures. 


Yes [_| No [| 


3. State and local staff have been trained on data entry into the state or local MIS 


Yes [_] No [| 


4. Local staff has had training on how to produce and/or interpret reports produced by the 
MIS. Yes [_] No [| 


5. Training has been provided on conducting follow-up survey or data matching procedures, 
to state or local staff involved in survey or matching. Yes [|_| No [] 


6. State has trained staff on distance education policy and use of proxy hours, if estimated. 


Yes [ | No [_] 
7. State provides at least one additional training annually to local programs on NRS issues, 
MIS data entry or data analysis issues. Yes] NoL] 


7a. If yes, briefly describe when the most recent additional training occurred, its duration 
and about what percent of local providers attended. This training should not be the same 
as the one described above in item number 1. 
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Superior Quality 


1. There is planned, continuous training (at least one training annually) on data collection 
and NRS issues. Yes [|] No [] 


la. If yes, briefly describe frequency, duration and content of trainings. 


2. NRS training is planned and delivered based on needs of local staff and evaluations of 
previous trainings. Yes[_] NoL] 


2a. If yes, briefly describe your needs assessment process. 


3. State has ongoing technical support to local programs to improve data matching and/or 
survey follow-up procedures, such as collecting the data. 
Yes[ |] NoL] 


3a. If yes, describe support and how it is provided. 


Exemplary Quality 


1. State has developed and is implementing a plan for ongoing staff development on NRS 
and data use issues to promote continuous improvement. Yes L_] No L] 


la. If yes, briefly describe the plan. 


2. State has a system for continuous training of local staff on NRS issues, data collection 
and data reporting through regularly scheduled training sessions or other resources. 


Yes [|] No [ | 


3. State has timely intervention strategies to identify data problems as they occur and to 
provide training to programs to correct the problems. Yes [_] No L_] 


3a. If yes, briefly describe the process. 
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Za 
NRS 


National Reporting System 
for Adult Education 


A Project of the U.S. Department of Education 


DATA QUALITY IMPROVEMENT PLAN 


The state must submit a quality improvement plan for each content area that does not meet all 
of the standards within the superior level. A separate plan must be completed for each content area. 
The plans should not exceed one page and include the following information. 


1. Content area (e.g., Data Foundation and Structure, Staff Development) and specific 
standard(s) not met. 


2. For each standard not met, describe your planned approach to implementing changes that 
will allow you to meet the standard. 


3. Describe the barriers or problems you anticipate, if any, to implement these plans. 


4. Describe any technical assistance you might need to implement these planned changes. 


5. If you believe you will be unable to meet any standard please explain why. 
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<A 
NRS 


National Reporting System 
for Adult Education 


A Project of the U.S. Department of Education 


NRS DATA QUALITY CHECKLIST CERTIFICATION 


Note: The state director of adult education or head of the state administrative agency in which 
the federal adult education program resides must sign this certification. 


CERTIFICATION 


I certify that to the best of my knowledge, the information contained in this document is true and 
correct and accurately reflects the state’s data collection policies and procedures for collecting and 
reporting data for the U.S. Department of Education’s National Reporting System for adult 
education. 


Signature 


Name and Title 


Date 


Seal 
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REPORTING TABLES 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a 
collection of information unless such collection displays a valid OMB control number. The valid 
OMB control number for this information collection is 1830-0027. The time required to complete 
this information collection is estimated to average 120 hours per response, including the time to 
review instructions, search existing data resources and gather the data needed, and complete and 
review the information collection. If you have any comments concerning the accuracy of the 
time estimate or suggestions for improving this form, please write to: Division of Adult 
Education and Literacy, Office of Vocational and Adult Education, U.S. Department of Education, 
400 Maryland Avenue, S.W., Washington, DC 20202-4651. If you have comments or concerns 
regarding the status of your individual submission of this form, write directly to: Division of 
Adult Education and Literacy, Office of Vocational and Adult Education, U.S. Department of 
Education, 400 Maryland Avenue, S.W., Washington, DC 20202-4651. 
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Table 1 
Participants by Entering Educational Functioning Level, Ethnicity, and Sex 


Enter the number of participants* by educational functioning level,** ethnicity,“* and sex. 


Entering Educational 
Functioning Level 
(A) 
ABE Beginning Literacy 


ABE Beginning Basic Education 
ABE Intermediate Low 

ABE Intermediate High 

ASE Low 

ASE High 

ESL Beginning Literacy 


ESL Low Beginning 
ESL High Beginning 


ESL Intermediate Low 


ESL Intermediate High 
ESL Advanced 


Total 


OMB Number 1830-0027, Expires 08/31/2017. 

“A participant is an adult who receives at least twelve (12) hours of instruction. Work-based project learners are not included in this table. 

™See definitions for educational functioning levels. 

*** See definitions of race/ethnicity categories and examples that demonstrate how to report them. A participant should be included in the racial/ethnic group to which he or she appears to belong, 
identifies with, or is regarded in the community as belonging. If a student does not self-identify a race/ethnicity, the program must use observer identification. 
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Table 2 
Participants by Age, Ethnicity, and Sex 


Enter the number of participants by age,* ethnicity,** and sex. 


American Indian 


Male Female Male Female Male Female Male Female 


60 and Older 


= ee a a a (a a | ee as 


*Participants should be classified based on their age at entry. Participants entering the program prior to the current program year should be classified based on their age at the beginning of the 
current program year. Work-based project learners are not included in this table. 

** See definitions of race/ethnic categories and examples that demonstrate how to report them. 

The totals in Columns B-Oshould equal the totals in Column B-Oof Table 1. Row totals in Column Pshould equal corresponding column totals in Table 3. 

OMB Number 1830-0027, Expires 08/31/2017. 
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Table 3 
Participants by Program Type and Age 


Enter the number of participants by program type and age. 


(A) 


Adult Basic Education 
Adult Secondary Education 


English-as-a-Second Language 


The total in Colum G should equal the total in Column P of Table 1. 


OMB Number 1830-0027, Expires 08/31/2017. 
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Table 4 
Educational Gains and Attendance by Educational Functioning Level 


Enter number of participants for each category listed, total attendance hours, and calculate percentage of participants completing each level. 


Entering Educational 
Functioning Level 


(A) 


ASE Low 


ESL Beginning Literacy 
ESL High Beginning 


ESL Intermediate High 
ESL Advanced 


LT 


Total 


The total in Columns 8-F should equal the total for the corresponding rows in Column Pof Table 2. 

Column D is the total number of learners who completed a level, including learners who left after completing and learners who remained enrolled and moved to one or more higher levels. 
Column Erepresents a subset of Column D(Number Completed Level) and is learners who completed a level and enrolled in one or more higher levels. 

Column Fis students who left the program or received no services for 90 consecutive days and have no scheduled services. 

Column D+ F+Gshould equal the total in Column B 

Column Grepresents the number of learners still enrolled who are at the same educational level as when they entered. 

ColumnD 


Each row total in Column His calculated by using the following formula: H = 
ColumnB 


Work-based project learners are not included in this table. 
*Completion of ASE high level is attainment of a secondary credential or passing a State recognized high school equivalency test. 
OMB Number 1830-0027, Expires 08/31/2017. 
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Table 4B 
Educational Gains and Attendance for Pre- and Posttested Participants 


Enter number of pre- and posttested participants for each category listed, calculate percentage of posttested participants completing each level, and enter total 
attendance hours for posttested completion. 


ABE Beginning Literac 
ABE Beginning Basic Education 
ABE Intermediate Low 
ABE Intermediate High 
ASE Low 

ASE High* 

ESL Beginning Literacy 
ESL Low Beginning 
ESL High Beginning 
ESL Intermediate Low 
ESL Intermediate High 
ESL Advanced 


(A) 


Include in this table only students who are both pre- and posttested. 
Column Dis the total number of learners who completed a level, including learners who left after completing and learners who remained enrolled and moved to one or more higher levels. 
Column Erepresents a subset of Column D(Number Completed Level) and is learners who completed a level and enrolled in one or more higher levels. 
Column Fis students who left the program or received no services for 90 consecutive days and have no scheduled services. 
Column D+F +Gshould equal the total in Column B 
Column Grepresents the number of learners still enrolled who are at the same educational level as when they entered. 
ColumnD 


Each row total in Column His calculated by using the following formula: H = 
ColumnB 


Work-based project learners are not included in this table. 
*Completion of ASE high level is attainment of a secondary credential or passing a State recognized high school equivalency test. 
OMB Number 1830-0027, Expires 08/31/2017. 
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Table 4C 
Educational Gains and Attendance for Participants in Distance Education 


Enter number of distance education participants for each category listed, calculate percentage of participants completing each level, and enter total proxy and direct 


attendance hours. 


Entering Educational 
Functioning Level 
(A) 
ABE Beginning Literac 
ABE Beginning Basic Education 
ABE Intermediate Low 
ABE Intermediate High 


ASE Low 

ASE High* 

ESL Beginning Literacy 
ESL Low Beginning 


ESL High Beginning 
ESL Intermediate Low 


ESL Intermediate High 
ESL Advanced 


Include in this table only students who are counted as distance education students. 
Column D is the total number of learners who completed a level, including learners who left after completing and learners who remained enrolled and moved to one or more higher levels. 
Column E represents a subset of Column D (Number Completed Level) and is learners who completed a level and enrolled in one or more higher levels. 
Column F is students who left the program or received no services for 90 consecutive days and have no scheduled services. 

Column D + F + G should equal the total in Column B. 

Column G represents the number of learners still enrolled who are at the same educational level as when they entered. 


Each row total in Column His calculated by using the following formula: H = 


ColumnD 
ColumnB 


Work-based project learners are not included in this table. 
*Completion of ASE high level is attainment of a secondary credential or passing a State recognized high school equivalency test. 
OMB Number 1830-0027, Expires 08/31/2017. 
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Table 5 
Core Follow-up Outcome Achievement 
M Number of 
e Number of | Participants | Response | Number of | Number of 
t Participants | Responding Rate or Participants | Participants | Percent 
h| Number of Used for to Survey or | Percent Achieving Achieving Achieving 
Core Follow-up 0 | Participants |Representative) Available for | Available Outcome Outcome Outcome 
Outcome Measures /|d|_ in Cohort Cohort Data Matching] for Match | (Unweighted) | (Weighted) | (Weighted) 
(A) (B) (C) (D) (E) (F) (G) (H) 
Entered Employment* | U N/A 
R 
C 
Retained U N/A 
Employment** 
R 
C 
Obtained a Secondary |U N/A 
School Credential or Its 
Equivalent *** R 
C 
Entered Postsecondary | U N/A 
Education or Training — 
Current program R 
year**** 
C 
Entered Postsecondary | U N/A 
Education or Training — 
prior program year**** [R 
C 


U = Universe cohort; programs attempted to collect data for all eligible participants either by survey, data 
match or both 

R = Representative cohort; programs attempted to collect data for a representative subset of eligible 
participants 

C = Combined universe and representative cohort totals; these numbers will be calculated automatically 
by OCTAE’s data system 
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Instructions for Completing Table 5 


Note: All shaded columns (E, G, and H) and rows for cohort totals will be calculated automatically by 
OCTAE’s data system. 


* Report in Column B the number of participants who were unemployed at entry and in the labor force who 
exited during the program year. Do not exclude students because of missing Social Security numbers or 
other missing data. 


*™ Report in Column B: (1) the number of participants who were unemployed at entry and who were in the 
labor force who exited during the program year and who entered employment by the end of the first quarter 
after program exit and (2) the number of participants employed at entry who exited during the program 
year. 


** Report in Column B the number of participants who (1) took all subject areas of a State recognized high 
school equivalency test or (2) were enrolled in adult high school at the high ASE level or (3) were enrolled 
in the assessment phase of the EDP, who exited during the program year. 


** Report in Column B the number of participants who (1) have earned high school diploma or its 
equivalent while enrolled in adult education, or (2) have a secondary credential at entry, or (3) are enrolled 
in aclass specifically designed for transitioning to community college, who exited during the program year. 
Enter the number of these participants who enrolled during the current program year in the row labeled 
current program year. Enter the number of these participants who enrolled during the program year 
immediately prior to the current year in the row labeled prior program year. However, this row is not to be 
completed until the PY 2013 report. Leave blank for PY 2012 reporting. 


For Columns B through D and Column F, enter the information separately for programs that attempted to 
collect data from all eligible participants in each cohort (i.e., the universe cohort) and for programs that 
used representative cohorts. The first row for each follow-up outcome measure should be used to report 
information based on universe cohorts, and the second row should be used to report information based on 
representative cohorts. If no programs used representative cohorts, the “R” rows should be left blank. 


If survey is used, then the number in Column D should be less than Column C, unless there was a 100- 
percent response rate to the survey. If data matching is used, then the number reported in Column D 
should be the total number of records available for the data match. That number is normally less than the 
number in Column B. (If the numbers in these two columns are equal, then it means that all Social Security 
numbers are valid and that there are no missing Social Security numbers.) 


Column E= comma? , for programs that used universe cohorts. 


olumnB 

ColumnD 

ColumnC | 

Column H is the number in Column G divided by the number in Column B. Column H should never be 
greater than 100 percent. If the response rate is less than 50 percent (Column E) for universe cohorts or 
less than 70 percent for representative cohorts, then the data system will not return a valid percent in 
Column H. 


For programs using representative cohorts, Column E = 


OMB Number 1830-0027, Expires 08/31/2017. 
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U = Universe cohort; programs attempted to collect data for all eligible participants either by survey, data 


match or both 


R = Representative cohort; programs attempted to collect data for a representative subset of eligible 


participants 


C = Combined universe and representative cohort totals; these numbers will be calculated automatically 


by OCTAE’s data system 


Table 5A 
Core Follow-up Outcome Achievement for Participants in Distance Education 
M Number of 
e Number of | Participants | Response | Number of | Number of 
t Participants | Responding Rate or Participants | Participants | Percent 
h| Number of Used for to Survey or | Percent Achieving Achieving Achieving 
Core Follow-up 0 | Participants |Representative) Available for | Available Outcome Outcome Outcome 
Outcome Measures /|d|_ in Cohort Cohort Data Matching] for Match | (Unweighted) | (Weighted) | (Weighted) 
(A) (B) (C) (D) (E) (F) (G) (H) 
Entered Employment* | U N/A 
R 
C 
Retained U N/A 
Employment** 
R 
C 
Obtained a Secondary |U N/A 
School Credential or Its 
Equivalent *** R 
C 
Entered Postsecondary | U N/A 
Education or Training — 
Current program R 
year**** 
C 
Entered Postsecondary | U N/A 
Education or Training — 
prior program year**** [R 
C 
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Instructions for Completing Table 5A 
Include in this table only students who are counted as distance education students. 
Follow the same instructions for Completing Table 5 to complete Table 5a, repeated below. 


Note: All shaded columns (E, G, and H) and rows for cohort totals will be calculated automatically by 
OCTAE’s data system. 


* Report in Column B the number of participants who were unemployed at entry and in the labor force who 
exited during the program year. Do not exclude students because of missing Social Security numbers or 
other missing data. 


™ Report in Column B: (1) the number of participants who were unemployed at entry and who were in the 
labor force who exited during the program year and who entered employment by the end of the first quarter 
after program exit and (2) the number of participants employed at entry who exited during the program 
year. 


** Report in Column B the number of participants who (1) took all subject areas of a State recognized high 
school equivalency test or (2) were enrolled in adult high school at the high ASE level or (3) were enrolled 
in the assessment phase of the EDP, who exited during the program year. 


** Report in Column B the number of participants who (1) have high school diploma or its equivalent while 
enrolled in adult education, or (2) have a secondary credential at entry, or (3) are enrolled in a class 
specifically designed for transitioning to community college, who exited during the program year. Enter the 
number of these participants who enrolled during the current program year in the row labeled current 
program year. Enter the number of these participants who enrolled during the program year immediately 
prior to the current year in the row labeled prior program year. However, this row is not to be completed 
until the PY 2013 report. Leave blank for PY 2012 reporting. 


For Columns B through D and Column F, enter the information separately for programs that attempted to 
collect data from all eligible participants in each cohort (i.e., the universe cohort) and for programs that 
used representative cohorts. The first row for each follow-up outcome measure should be used to report 
information based on universe cohorts, and the second row should be used to report information based on 
representative cohorts. If no programs used representative cohorts, the “R” rows should be left blank. 


If survey is used, then the number in Column D should be less than Column C, unless there was a 100- 
percent response rate to the survey. /f data matching is used, then the number reported in Column D 
should be the total number of records available for the data match. That number is normally less than the 
number in Column B. (If the numbers in these two columns are equal, then it means that all Social Security 
numbers are valid and that there are no missing Social Security numbers.) 


ColumnD 


Column E = ———., for programs that used universe cohorts. 
olumnB 
For programs using representative cohorts, Column E = folmmnD 
ColumnC | 


Column H is the number in Column G divided by the number in Column B. Column H should never be 
greater than 100 percent. If the response rate is less than 50 percent (Column E) for universe cohorts or 
less than 70 percent for representative cohorts, then the data system will not return a valid percent in 
Column H. 
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NRS Implementation Guidelines C-12 


Appendix C: NRS Reporting Tables 


Table 6 
Participant Status and Program Enrollment 


Enter the number of participants for each of the categories listed. 


Number 


Living in Rural Area* 


US Based Non-US Based 


i kk 
Highest Degree or Level of School Completed Sehoolinia Schooling 


No schooling 

Grades 1-5 

Grades 6-8 

Grades 9-12 (no diploma) 

High School Diploma or alternate credential 
High School Equivalency certificate 

Some college, no degree 


College or professional degree 


Unknown 
Program Type 
In Family Literacy Program** 
In Workplace Literacy Program** 
In Program for the Homeless** 


In Program for Work-based Project Learners** 


Institutional Programs 


In Other Institutional Setting 
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Secondary Status Measures (Optional) 


Learning Disabled Adult 


*Rural areas are places with less than 2,500 inhabitants and located outside urbanized areas. 
“Participants counted here must be in a program specifically designed for that purpose. 
*“Enter the highest level of schooling or degree attained for each student in US or non-us-based schooling. Provide 


only one entry per student. The total number of students reported here must be the same as the table total reported in 
Table 1, Column P. 
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NRS Implementation Guidelines C-14 


Appendix C: NRS Reporting Tables 


Table 7 
Adult Education Personnel by Function and Job Status 


Enter an unduplicated count of personnel by function and job status. 


Adult Education Personnel 
Total Number of Total Number of 
Function Part-time Personnel Full-time Personnel Unpaid Volunteers 


Ee 
Supervisory/Ancillary Services 
sqm | | | 
Stincnicory/Ancilaty Services 
|| (Of 
fLocalCounselors || 
[Local Paraprofessionals | | 
Local Teachers FP 
Kea 
In Adult Education 

| Lessthanoneyear | | 
| Onetoteeyears | | 
| Morethanthreeyears [| 
FTeacherCeriicaion ~<a | 
F Nocetifcaton | TO 
| Adult Education Certification | | 
| K12Certfcation FP 


Special EGucaton Cericaion | ———=+d| SSS 
OA 


In Column B, count one time only each part-time employee of the program administered under the Adult Education State Plan 
who is being paid out of Federal, State, and/or local education funds. 


In Column C, count one time only each full-time employee of the program administered under the Adult Education State Plan 
who is being paid out of Federal, State, and/or local education funds. 


In Column D, report the number of volunteers (personnel who are not paid) who served in the program administered under the 
Adult Education State Plan. 


Report adult education experience and certification for paid teachers only, not volunteers. The total number of teachers for 
which experience is reported must equal the total number of teachers reported in Columns B and C. 


For certification, report all certifications a teacher has. Multiple responses are allowed. Report teachers who lack certification in 
the “No Certification” category. 


The number of teachers reported must represent the number of teachers, both full and part-time, who worked in the 
State’s adult education programs at any time during the reporting period. 
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Outcomes for Adults in Family Literacy Programs (Optional) 


Table 8 


Enter the number of participants in family literacy programs for each of the categories listed. 


M Number of 
e Number of | Participants | Response | Number of | Number of 
t Participants | Responding Rate or Participants | Participants | Percent 
h| Number of Used for to Survey or | Percent Achieving Achieving | Achieving 
Core Follow-up 0| Participants |Representative| Available for | Available Outcome Outcome Outcome 
Outcome Measures |d) in Cohort Cohort Data Matching] for Match | (Unweighted) | (Weighted) | (Weighted) 
(A) (B) (C) (D) (E) (F) (G) (H) 
Completed Educational | U N/A 
Functioning Level* 
Entered Employment | U N/A 
R 
C 
Retained Employment | U N/A 
R 
C 
Obtained a Secondary | U N/A 
School Credential or 
Its Equivalent *** R 
C 
Entered U N/A 
Postsecondary 
Education or Training — 
current program year 
C 
Entered U N/A 
Postsecondary 
Education or Training- [R 
prior program year 
C 
NRS Implementation Guidelines C-16 


Appendix C: NRS Reporting Tables 


Increased Involvement | U N/A 
in Children’s Education 

R 

C 


Helped more 
frequently with school 


Increased contact 
with children’s 
teachers 


More involved in 
children’s school 


activities 
Increased Involvement | U N/A 
in Children’s Literacy 
Activities R 


Reading to children 


Visiting library 


Purchasing books or 
magazines 


For reporting completion of Educational Functioning Level: 


* Report in Column B for this row all family literacy program participants who received 12 or more hours of service. Column F should 
include all participants reported in Column B who advanced one or more levels. 


ColumnF 


Compute Column H for this row using the following formula: H =————— 
ColumnB 


For reporting Follow-up Measures: 


Follow instructions for completing Table 5 to report these outcomes. However, include only family literacy program participants in 
Table 8. 


Achievement of one or more of the increased involvement in children’s education or children’s literacy activities measures should be 
counted only once per participant. However, the specific outcome should be recorded in the subcategory and more than one outcome 
may be reported, so that the total for the three subcategories may be greater than the total reported for the overall category. For 
example, a participant who helped more frequently with schoolwork and increased contact with child’s teachers would be recorded in 
both categories but would be counted only once in the overall category of “increased involvement in children’s education.” 
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Table 9 


Outcomes for Adults in Workplace Literacy Programs (Optional) 


Enter the number of participants in workplace literacy programs for each of the categories listed. 


M Number of 
e Number of | Participants | Response | Numberof | Number of 
t Participants | Responding Rate or Participants | Participants | Percent 
h| Number of Used for to Survey or | Percent Achieving Achieving | Achieving 
Core Follow-up 0 | Participants /Representative| Available for | Available Outcome Outcome Outcome 
Outcome Measures jd) _ in Cohort Cohort Data Matching| for Match | (Unweighted) | (Weighted) | (Weighted) 
(A) (B) (C) (D) (E) (F) (G) (H) 
Completed an 
Educational 
Functioning Level* 
Entered Employment |U N/A 
R 
C 
Retained Employment | U N/A 
R 
C 
Obtained a Secondary | U N/A 
School Credential or 
Its Equivalent *** R 
C 
Entered U N/A 
Postsecondary 
Education or Training - R 
current program year 
C 
Entered U N/A 
Postsecondary 
Education or Training- | R 
prior program year 
C 
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For reporting completion of Educational Functioning Level: 


* Report in Column B for this row all workplace literacy program participants who received 12 or more hours of 
service. Column F should include all participants reported in Column B who advanced one or more levels. 


ColumnF 


Compute Column G for this row using the following formula: G = ————— 
ColumnB 


For reporting Follow-up Measures: 


Follow instructions for completing Table 5 to report the outcomes. However, include only workplace literacy program 
participants in Table 9. 
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Table 10 


Outcomes for Adults in Correctional Education Programs 


Enter the number of participants in correctional education programs for each of the categories listed. 


M Number of 
e Number of | Participants | Response | Numberof | Number of 
t Participants | Responding Rate or Participants | Participants | Percent 
h| Number of Used for to Survey or | Percent Achieving Achieving | Achieving 
Core Follow-up 0 | Participants /Representative| Available for | Available Outcome Outcome Outcome 
Outcome Measures jd) in Cohort Cohort Data Matching| for Match | (Unweighted) | (Weighted) | (Weighted) 
(A) (B) (C) (D) (E) (F) (G) (H) 
Completed an 
Educational 
Functioning Level* 
Entered Employment |U N/A 
R 
C 
Retained Employment | U N/A 
R 
C 
Obtained a Secondary | U N/A 
School Credential or 
Its Equivalent *** R 
C 
Entered U N/A 
Postsecondary 
Education or Training - R 
current program year 
C 
Entered U N/A 
Postsecondary 
Education or Training- | R 
prior program year 
C 
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For reporting completion of Educational Functioning Level: 


* Report in Column B for this row all correctional educational program participants who received 12 or more hours of 
service. Column F should include all participants reported in Column B who advanced one or more levels. 


ColumnF 


Compute Column G for this row using the following formula: G = 
ColumnB 


For reporting Follow-up Measures: 


Follow instructions for completing Table 5 to report the outcomes. However, include only correctional educational 
program participants in Table 10. 


OMB Number 1830-0027, Expires 08/31/2017. 


NRS Implementation Guidelines C-21 


Appendix C: NRS Reporting Tables 


Table 11 
Secondary Outcome Measures (Optional) 


Enter the number of participants for each of the categories listed. 


Number of 
Number of Participants Percentage 
Participants Obtaining Achieving 
Secondary Outcome Measures Outcome Outcome 
A 


Increased Involvement in Children’s Literacy 


Each row total In Column D is calculated using the following formula: D = 


ColumnC 
ColumnB 


* Enter the total number of participants who achieved this outcome regardless of whether the participant was in a family literacy 
program. Use Table 8 to enter achievements of family literacy participants. The number reported here may be higher than 
reported in Table 8 because it includes all participants who achieved this goal. 
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Table 12 (Optional) 
Work-based Project Learners by Age, Ethnicity, and Sex 


Enter the number of work-based project learners by age,* ethnicity, and sex. 


Only participants designated as work-based project learners should be included in this table. These participants should not be included in Tables 1-5. 


The total in Column Nshould equal the number of work-based project learners reported in Table 6. 
*Participants should be classified based on their age at entry. 
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Table 14 


Local Grantees by Funding Source 


Enter the number of each type of grantee (see attached definitions) directly funded by the state and the 


amount of federal and state funding they receive. 


Total Total WIA Funding State Funding 
Number of Number of 
Providers Sub- 
Provider Agency (B) Recipients Total % of Total Total % of Total 
(A) (C) (D) (E) (F) (G) 


Local Education Agencies 


Public or Private Nonprofit Agency 


Community-based Organizations 


Faith-based Organizations 


Libraries 


Institutions of Higher Education 


Community, Junior or Technical 
Colleges 


Four-year Colleges or Universities 


Other Institutions of Higher 
Education 


Other Agencies 


Correctional Institutions 


Other Institutions (non-correctional) 


All Other Agencies 


Total 


1. In Column (B), report the number of providers receiving a grant award or contract for instructional services from the eligible agency. 
2. In Column (C), report the total number of each entity receiving funds as a sub-recipient. (Entities receiving funds from a grantee as part of a consortium are to 


reported in column (C). 
3. In Column (E), the percentage is to be calculated using the following formula: 


Column D 


Total WIA 


-- = Col(E) 


4. In Column (F), report total amount of state funds contributed. This amount need not necessarily equal the non-federal expenditure report on the Financial 


Status Report. 


5. In Column (G), the percentage is to be calculated using the following formula: Column F 


Total State 
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Grantee Definitions for Table 14 


Local Education Agencies are publicly funded entities designated to administer and provide primary and secondary 
education instruction and services within a city, county, school district, township or region. 


Community-based Organizations (CBOs) are private nonprofit organizations of demonstrated effectiveness that 
are representative of a community or significant segment of a community. 


Faith-based Organizations (FBO) are non-profit organizations associated with a faith community or multiple faith 
ministries. 


Libraries are public state and community funded institutions that offer education and community services in addition 
to providing access to print, audio-visual and technology resources. 


Community, Junior or Technical Colleges are public institutions of higher education that offer associate’s degree 
and certificate programs but, with few exceptions, award no baccalaureate degrees. 


Four Year Colleges or Universities are a public or private non-profit institution of higher education that primarily 
offers baccalaureate degree programs. 


Other Institution of Higher Education is a public or private non-profit institution that is not a community, junior, or 
technical college or a four-year college or university. 


Correctional Institutions refer to state or federal penal institutions for criminal offenders. These include prisons, 
jails, and other correctional detention centers. 


Other Institutions (Non-Correctional) are any medical or special institutions not designed for criminal offenders. 
All Other Agencies include other public (federal, state, local) agencies not listed in the categories above. 
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Narrative Report 
Instructions 


These instructions provide descriptive information for the narrative items in the report. The 
maximum number of pages is 10. Please use the following outline in preparing the narrative part of 
the Annual Performance Report: 


1. State Leadership Funds 
Describe the major activities supported with State Leadership funds during the reporting 


period and the extent to which these activities related to the implementation of career 
pathways. 


2. Performance Data Analysis 
Describe any significant findings from the evaluation of your performance data for the 
reporting period and efforts to improve outcomes for the core indicators of performance. 


3. Integration with Other Programs 
Describe how you have integrated activities funded under the Adult Education and Family 


Literacy Act (AEFLA) with other adult education, career development, and employment and 
training activities. Describe your efforts to develop or advance career pathway systems that 
include career and technical education, postsecondary education, employers, and economic 
and workforce development. Include a description of how your agency is being represented 
on the State and Local Workforce Investment Boards, the provision of core and other 
services through the One-Stop system, and an estimate of the AEFLA funds being used to 
support activities and services through the One-Stop delivery system. 


4. English Literacy/Civics (EL/C) Program 
Describe the activities and services supported with EL/C funds, including the number of 
programs receiving EL/C grants and an estimate of the number of adult learners served. 
(Only states receiving EL/C funds should respond to question #4). 


5. Secondary School Credentials and Equivalencies 
Describe your state’s policies related to how it awards a secondary school diploma or its 
equivalent to individuals who are no longer enrolled or required to be enrolled in secondary 
school under State law. Include state recognized tests that are used to award the diploma as 
well as other criteria that may be recognized, such as competency-based or credit-based 
methods. 


6. Adult Education Standards 
Describe your state’s progress toward implementation of adult education college and career 
readiness (CCR) standards, including whether your state has formally adopted a validated set 
of CCR standards. Describe how the state has determined the standards to be representative 
of CCR and the timeline by which such standards will be or have been implemented by all 
local programs. Describe how the state is supporting the use of standards by local programs 
and State Leadership funds that are being used to support implementation. 
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INSTRUCTIONS 
Federal Financial Report 
OMB Number 1830-0027 


U.S. Department of Education 
Office of Vocational and Adult Education 
Adult Education and Family Literacy Act of 1998 
Basic Grants to States—CFDA 084.002 


The new Federal Financial Report (FFR) must be used beginning in December 2013. A 
separate FFR is to be used for each Federal Funding Period as reported in Block 8 of the 
FFR for Adult Education. 


Instructions 


Federal Agency and 
Organizational 

lement to Which 
Report is Submitted 
Federal Grant or Other 
Identifying Number nter the PR/Award number as indicated in Block 2 of the 
Assigned by Federal (Grant Award Notification. 


A 


ea cs nter the name and complete address of the recipient 
3 Recipient Organization gE ees ; : 
organization including zip code. 
nter the recipient organization’s Data Universal Numbering 
A DUNS Number System (DUNS) number or Central Contract Registry 
extended DUNS number. 
nter the recipient organization’s Employer Identification 
A IN 
umber (EIN). 


nter the account number or any other identifying number 
.¢_:_ jassigned by the recipient to the award. This number is for the 
umber or Identifying | .°. ‘ : 
ete recipient’s use only and is not required by the Federal 
agency. 


6 Report Type ——_—__ [Indicate whether this FFR is an initial or final report. 


his block is preprinted. 


Recipient Account 
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Specify whether a cash or accrual basis was used for 
recording transactions related to the award and for preparing 
Basis of Accounting {this FFR. Accrual basis of accounting refers to the 
(Cash/Accrual) accounting method in which expenses are recorded when 
incurred. For cash basis accounting, expenses are recorded 
hen they are paid. 


nter the beginning and ending dates of the period in which 
lyou are reporting the financial activity of the grant. A first 
Reporting Period End _fyear report will cover the first 15 months of the grant period 
Date: (Month, Day, _ e.g., July 1, 2012 through September 30, 2013. The final 
report will cover the entire 27 months, which grantees have 
o obligate their funds e.g., July 1, 2011 through September 
B0, 2013. 


ransactions: Enter cumulative amounts from date of the inception of the award 
hrough the end date of the reporting period specified in line 9. 


nter the cumulative amount of Federal fund disbursements 
(such as cash or checks) as of the reporting period end date. 
Disbursements are the sum of actual cash disbursements for 
Cash Disbursements (direct charges for goods and services, the amount of indirect 
expenses charged to the award, and the amount of cash 
advances and payments made to subrecipients and 
contractors. 
nter the amount of Line 10a minus Line 10b. This amount 
represents immediate cash needs. If more than three business 
Cash On Hand (Line _ days of cash are on hand, the Federal agency may require an 
10a Minus Line 10b _ explanation on Line 12, Remarks, explaining why the 
drawdown was made prematurely or other reasons for the 
excess cash. 


otal Federal Funds nter the total Federal funds authorized as of the reporting 
10d : ; 
Authorized period end date. 
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Federal Share of 
xpenditures 


Federal Share of 


otal Federal Share 
(Sum of Lines 10e and 


lof Federal Funds (Line 
10d Minus Line 10g) 


otal Recipient Share 
Required 


nter the amount of Federal fund expenditures. For reports 
prepared on a cash basis, expenditures are the sum of cash 
disbursements for direct charges for property and services; 
he amount of indirect expense charged; and the amount of 
cash advance payments and payments made to subrecipients. 
For reports prepared on an accrual basis, expenditures are the 
Sum of cash disbursements for direct charges for property 
and services; the amount of indirect expense incurred; and 
he net increase or decrease in the amounts owed by the 
recipient for (1) goods and other property received; (2) 
Services performed by employees, contractors, subrecipients, 
and other payees; and (3) programs for which no current 
Services or performance are required. 

nliquidated obligations on a cash basis are obligations 
incurred, but not yet paid. On an accrual basis, they are 
obligations incurred, but for which an expenditure has not 
lyet been recorded. Enter the Federal portion of unliquidated 
obligations. Those obligations include direct and indirect 
expenses incurred but not yet paid or charged to the award, 
including amounts due to subrecipients and contractors. On 
he final report, this line should be zero unless the awarding 
agency has provided other instructions. Do not include any 
amount in Line 10f that has been reported in Line 10e. Do 
not include any amount in Line 10f for a future commitment 
of funds (such as a long-term contract) for which an 
obligation or expense has not been incurred. 


nter the sum of Lines 10e and 10f. 


nter the amount of Line 10d minus Line 10g. 


nter the total required recipient share (i.e. the greater of 
required match or maintenance of effort) for reporting period 
specified in line 9. The required recipient share should 
include all matching and cost sharing provided by recipients 
and third-party providers to meet the level required by the 
Federal agency. 

nter the recipient share of actual cash disbursements or 
outlays (less any rebates, refunds, or other credits) including 
payments to subrecipients and contractors. This amount may 
include the value of allowable third party in-kind 
contributions. Note: On the final report this line should be 
equal to or greater than the amount of Line 10i. 
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Remaining Recipient 

Share to be Provided 

(Line 10i Minus 
ine10j) 


otal Program Income : 
nter the amount of program income earned. 


nter the amount of program income that was added to funds 
committed to the total project costs and expended to further 
eligible project or program activities. 


nter the amount of Line 10i minus Line 10}. If recipient 
share in Line 10j is greater than the required match amount 
in Line 10i, enter zero. 


Program Income 
xpended 


nexpended Program {Enter the amount of Line 101 minus Line 10m or Line 10n. 
Income (Line 101 his amount equals the program income that has been earned 
inus Line 10m) but not expended, as of the reporting period end date. 
Indirect Expense: Enter cumulative amounts from date of the inception of the award 
hrough the end date of the reporting period specified in line 9. 


nee wn Indicate whether indirect cost rate is Restricted Provisional or 
yp Restricted Final. 


lib Rae ss Rae Enter the indirect cost rate in Enter the indirect cost rate in effect during the reporting period. Enter the indirect cost rate in effect during the reporting period. the reporting period. 


aT From; Period To [Enter the beginning and ending effective dates for the rate. 


lid Bae si ase Enter the amount Enter the amount of the base against which the rate was applied] ue bee against which ane rate was Enter the amount of the base against which the rate was applied 


= fe 
: eriod specified. (Multiply 11b. x 11d. ) 


Remarks Enter any explanations or additional information required by 
he Federal sponsoring agency including excess cash as stated 
in line 10c. 


yped or Printed Name [Enter the name and title of the authorized certifying official. 
and Title of Authorized 
Certifying Official 


Certifying Official 
INumber and Extension) 2 the individual listed in Line 13a. 
i cml ee acl 
(Month, Day, Year) | ducation using the month, day, year format. 
fener 
Agency Use Only 
agency use. 


Federal Financial Reports are due on December 31 of each year. Reports must be submitted 
electronically via the online NRS reporting system. In addition, a signed copy must be scanned 
and sent by email to the address provided in the DAEL annual reporting memo. 
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1. Federal Agency and Organizational Element to Which |2. Federal Grant or Other Identifying Number Assigned by Federal Agency 


FEDERAL FINANCIAL REPORT Report is Submitted 
TOTAL ALLOCATION U.S. Department of Education 


Division of Adult Education and Literacy 


OMB Approval Number: 1830-0027 
Expiration Date: 8/31/2014 


3. Recipient Organization (Name and complete address including Zip code) 4a. DUNS Number Ab. EIN 5. Recipient Account Number or Identifying Number 6. Report Type 7. Basis of Accounting 
Initial Cash 
OO Final C Accrual 
8. Project/(Grant Period (Month, Day, Year) 9. Reporting Period End Date (Month, Day, Year) 
From: To: From: To: 
10. Transactions (a) State Administration (b) State Leadership (c) Programs of Instruction (d) Programs of Instruction (e) Institutionalized Persons (f) Total 


(0-8) (9-12) 


Federal Cash: 
a. Cash Receipts 


b. Cash Disbursements 


c. Cash on Hand (line a minus b) 


Federal Expenditures and Unobligated Balance: 


eral funds authorized 


e. Federal share of expenditures 


f. Federal share of unliquidated obligations 


Fe 


leral share (sum of lines e and f) 


h. Unobligated balance of Federal funds (line d minus g) 


Recipient Share: 


i. Total recipient share required (i.e. Maintenance of Effort) 


j. Recipient share of expenditures 


k. Remaining recipient share to be provided (line i minus j) 


Program Income: 


|. Total program income earned 


m. Program income expended 


n. Unexpended program income (line | minus line m or line n) 


11. a. Type b. Rate c. Period From Period To d. Base 
Indirect Restricted Provisional 
Expense 

Restricted Final 


12. Remarks: Attach any explanations deemed necessary or information required by Federal sponsoring agency in compliance with governing legislation: 


13. Certification: By signing this report, | certify to the best of my knowledge and belief that the report is true, complete, and accurate, and the expenditures, disbursements and cash receipts are for the purposes and intent set forth in the award documents. | am 


aware that any false, fictitious, or fraudulent information may subject me to criminal, civil, or administrative penalties. (U.S. Code, Title 18, Section 1001) 


a. Typed or Printed Name and Title of Authorized Certifying Official c. Telephone (Area code, number, and extension) 


d. Email Address 


b. Signature of Authorized Certifying Official e. Date Report Submitted (Month, Day, Year) 


14. Agency use only: 


C-33 
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Financial Reporting Requirements for EL-Civics Funding 


States expending EL-Civics funds under the conditions outlined in Program Memorandum 2000-19, 
issued by Ronald S. Pugsley on May 16, 2000, shall report those expenditures as follows: 


In addition to submitting an annual FFR reporting all Federal and non-Federal expenditures, 
including those for EL-Civics, a separate FFR for EL-Civics expenditures is also required. This EL- 
Civics FFR, which represents a sub-total of the overall report, will provide the necessary information 
to determine that EL-Civics expenditures were in compliance with existing statutory requirements. A 
specially identified EL-Civics FFR is included for your use. 
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FEDERAL FINANCIAL REPORT 
EL / Civics 


1. Federal Agency and Organizational Element to Which 
Report is Submitted 


U.S. Department of Education 
Division of Adult Education and Literacy 


2. Federal Grant or Other Identifying Number Assigned by Federal Agency 


OMB Approval Number: 1830-0027 
Expiration Date: 8/31/2014 


3. Recipient Organization (Name and complete address including Zip code) 


4a. DUNS Number Ab. EIN 5. Recipient Account Number or Identifying Number 


6. Report Type 


LI Initial 


Final 


7. Basis of Accounting 


[J cash 
Accrual 


8. Project/Grant Period (Month, Day, Year) 


From: From: 


10. Transactions 


Federal Cash: 


a. Cash Receipts 


9. Reporting Period End Date (Month, Day, Year) 


(a) State Administration 


(b) State Leadership (c) Programs of Instruction 


(0-8) 


(d) Programs of Instruction 
(9-12) 


(e) Institutionalized Persons 


(f) Total 


b. Cash Disbursements 


c. Cash on Hand (line a minus b) 


Federal Expenditures and Unobligated Balance: 


d. Total Federal funds authorized 


e. Federal share of expenditures 


f. Federal share of unliquidated obligations 


g. Total Federal share (sum of lines e and f) 


h. Unobligated balance of Federal funds (line d minus g) 


Recipient Share: 


i. Total recipient share required (i.e. Maintenance of Effort) 


j. Recipient share of expenditures 


k. Remaining recipient share to be provided (line i minus j) 


Program Income: 


|. Total program income earned 


m. Program income expended 


n. Unexpended program income (line | minus line m or line n) 
11. a. Type b. Rate c. Period From 
Indirect Restricted Provisional 
Expense 
Restricted Final 


d. Base 


Period To e. Amount Charged 


g. Totals: 


12. Remarks: Attach any explanations deemed necessary or information required by Federal sponsoring agency in compliance with governing legislation: 


13. Certification: By signing this report, | certify to the best of my knowledge and belief that the report is true, complete, and accurate, and the expenditures, disbursements and cash receipts are for the purposes and intent set forth in the award documents. | am 
aware that any false, fictitious, or fraudulent information may subject me to criminal, civil, or administrative penalties. (U.S. Code, Title 18, Section 1001) 


a. Typed or Printed Name and Title of Authorized Certifying Official 


c. Telephone (Area code, number, and extension) 


d. Email Address 


b. Signature of Authorized Certifying Official 


e. Date Report Submitted (Month, Day, Year) 


14. Agency use only: 
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Comments and Responses on Proposed Educational Functioning Level Descriptors for 
Adult Basic Education 


Comment: 


We received several comments suggesting changes to the National Reporting System not related 
to the proposed revisions to the Educational Functioning Level (EFL) descriptors that we 
submitted for public comment. Two commenters suggested revising the exit-based criteria for 
the transition to postsecondary education and training performance indicator claiming that the 
requirement penalizes integrated education and training programs now codified under the 
Workforce Innovation and Opportunity Act (WIOA). Two commenters suggested that we 
eliminate the EFL descriptors and use only standardized test scale scores to report the 
educational gain performance indicator. One commenter suggested that we allow for reporting 
educational gains in more than one content domain (for example, math and reading rather than 
math or reading). One commenter suggested that we count students at all EFLs who pass high 
school equivalency examinations as having met the educational gain performance indicator 
rather than only students at the high adult secondary education EFL. Another commenter 
suggested that we consider the fact that legislatively mandated core indicators of performance 
and established GPRA measures are not necessarily representative of students’ goals for 
participating in the program. One commenter suggested that we create an independent 
framework for work readiness and cross-disciplinary competencies. The same commenter 
suggested that we use WIOA implementation as an opportunity to update the NRS. Another 
commenter suggested that we expand the scope of the NRS. The commenter suggested that we 
expand the assessment framework of the NRS by developing an additional, independent 
framework for work readiness and cross-disciplinary standards (e.g. problem solving, intellectual 
curiosity, work habits). The commenter felt that such a framework could provide a more flexible 
mechanism for describing student performance independently at any educational level and serve 
as an asset for WIOA partners working together to support successful outcomes for workforce 
development system customers. 


Discussion: 


The above comments focus broadly on changes to the NRS and do not address specific changes 
to the proposed EFL descriptors published in this notice. The Department appreciates the 
commenters’ interests in enhancing the NRS and fully intends to use the Workforce Innovation 
and Opportunity Act, final regulations, information collection requests and non-regulatory 
guidance to maximize the utility of the NRS in supporting successful WIOA implementation. 


Changes to Proposed EFL Descriptors: 
None 
Comment: 


We received several comments related to assessment of educational gain under the NRS. One 
commenter stated that assessments approved for use in the NRS to measure educational gain 
need to be better aligned to content standards and the proposed EFL descriptors. That same 
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commenter suggested that assessments approved for use in the NRS needed to be more 
computer-based/adaptive. Five other commenters disagreed with the suggestion that more 
assessments need to be computer-based/adaptive. Four commenters suggested that more 
assessments for English language learning domains (listening, speaking, reading, and writing) 
need to be approved for reporting educational gain in the NRS and the approval periods for tests 
need to be longer than some tests currently receive. One commenter inquired how college and 
career readiness will be assessed under WIOA. 


Discussion: 


The above comments focus broadly on assessment of educational gain in the NRS and do not 
address the proposed changes or revisions to the EFL descriptors as published in this notice. We 
appreciate commenters’ concerns for high quality assessment of educational gain in the NRS. 
We share those concerns and that is why we revised the EFL descriptors to better align with 
rigorous college and career readiness standards. Additionally, we recently submitted a Notice of 
Proposed Rulemaking (NPRM) in which we propose, among other things, updates to our 
regulations concerning assessment of educational gain in the NRS. As we move forward with 
implementation of the Workforce Innovation and Opportunity Act, we intend to continue to seek 
out appropriate Federal actions to support continued enhancement of assessment of learning in 
adult education. 


Changes to Proposed EFL Descriptors: 


None 
Comment: 


One commenter suggested that we revise our technical assistance publication, College and 
Career Readiness Standards for Adult Education, to have the same number of levels as the 
proposed EFL descriptors. The commenter stated that the College and Career Readiness 
Standards for Adult Education are divided into five instructional levels in each of the content 
domains while the proposed EFL descriptors were divided into six levels in each domain. The 
commenter felt that this difference between the College and Career Readiness Standards for 
Adult Education and the proposed EFL descriptors might result in adult educators having to 
guess in terms of what instructional activities are relevant and appropriate to particular EFLs. 


Discussion: 


The commenter did not suggest specific changes to the proposed EFL descriptors submitted for 
public comment. Rather, the commenter proposed a revision to the technical assistance 
publication, College and Career Readiness Standards for Adult Education. In developing that 
technical assistance publication, we worked closely with subject matter experts to convene two 
independent panels—one for English language arts and literacy (ELA/literacy) and one for 
mathematics—to look at the Common Core State Standards (CCSS) from the perspective of 
adult education. Each panel included a mix of expertise and experience, including representatives 
from adult education, community colleges, career and technical training, and the military. The 
methodology employed was deliberative, multilayered, iterative, and evidence-based. Over nine 
months, panelists were asked to make reasoned judgments about the relevance of the CCSS for 
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adults, based on where the evidence for college and career readiness was most compelling, and 
to revisit and verify those judgments in light of feedback and new questions. The decision to 
produce a publication which divided the standards into five rather than six levels was based upon 
the deliberations of these panels of experts and vetted with adult educators. 


Changes to Proposed EFL Descriptors: 


None 
Comment: 


We received several comments related to the increased academic rigor reflected in the proposed 
EFL descriptors. Several commenters stated that the increased rigor may pose particular 
challenges for specific subpopulations including English language learners and learning disabled 
individuals. Some commenters expressed concern that the increased rigor might result in a 
disincentive for students to enroll in adult education programs theorizing some students may 
become discouraged because they may not be able to move out of EFLs quickly. Other 
commenters stated that more rigorous EFLs might mean that students would not be able to show 
educational gains for NRS reporting purposes. One commenter stated that this could have a 
potential negative impact on local funding when that funding is based upon EFL advancement. 
Another commenter asked if we would institute a transition plan for moving toward greater rigor. 
Other commenters supported the increased rigor reflected in the proposed EFL descriptors noting 
that this increase would improve alignment with K — 12 education. Some commenters stated that 
the proposed EFL descriptors better align with the College and Career Readiness Standards for 
Adult Education. Three commenters stated that the increased rigor reflected in the proposed EFL 
descriptors would result in greater rigor in instruction and assessment thereby eliminating the 
need for subsequent remediation or transition classes for students who exited adult education at 
EFL 6. 


Discussion: 


We acknowledge that the proposed EFL descriptors reflect a greater degree of academic rigor 
than the current EFL descriptors and it may take time to adjust instructional practices to the 
increase in academic rigor. As we proceed in implementing revised EFL descriptors that reflect 
greater academic rigor, we continue to be mindful of the need to transition instruction at an 
appropriate pace providing adequate technical assistance along the way. We do not believe that 
high quality, evidence-based instruction focused on student success results in disincentives for 
individuals to participate. Nor do we believe that particular subpopulations should be exempted 
from the opportunity to access more rigorous instructional services that result in college and 
career readiness. Indeed, it is one of our strategic goals to improve the education system’s ability 
to consistently deliver excellent instruction aligned with rigorous academic standards while 
providing effective support services to close achievement and opportunity gaps, and ensure all 
students exit adult education programs college- and career-ready. This includes increasing 
educational opportunities for underserved students so that all students are well-positioned to 
succeed. We appreciate the acknowledgement and support for this goal that several commenters 
shared. 
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Changes to Proposed EFL Descriptors: 


None 
Comment: 


Several commenters expressed concern regarding the ability of English language learners 
enrolled in adult education programs to demonstrate educational gains within a reporting period 
based upon the proposed EFL descriptors. They stated that, in particular, low literate and non- 
literate English language learners would likely struggle to demonstrate educational gain 
according to the proposed EFL descriptors and cautioned that demonstration of educational gain 
against the proposed descriptors would likely take considerable time. 


Discussion: 


We support states’ adoption of rigorous college and career readiness standards designed to 
ensure that all adult education students can succeed in postsecondary education, training and the 
workforce. Further, we support the use of such standards for all students—including English 
language learners— to ensure that each student reaches his or her full potential. We seek to 
eliminate disproportionate educational and skill gaps currently prevalent among certain sub- 
populations. We acknowledge the linguistic and cultural diversity that English language learners 
contribute to adult education classrooms. We also realize that the social and academic uses of 
language inherent in — and needed to fully access — language-rich college-and-career-ready 
content standards may require that students and teachers look to additional English language 
proficiency (ELP) standards to support English language learners’ progress. Such ELP standards 
focus on the language needed to access college- and- career-ready standards in English Language 
Arts (ELA) and Literacy, Mathematics, and Science. It is our intention to identify and validate a 
set of ELP standards appropriate for use in adult education and to use those ELP standards to 
revise the current NRS EFL descriptors for English as a Second Language. It is not our intention 
to measure English language proficiency using the proposed EFL descriptors published in this 
notice, but rather to publish separate ELA descriptors to measure educational gain in the NRS at 
a future date. 


Changes to Proposed EFL Descriptors: 


None 
Comment: 


Several commenters discussed the clarity of the proposed EFL descriptors. One commenter 
expressed support for the descriptors as proposed stating that they provided clear distinctions 
between levels. Other commenters felt that we needed to include examples clearly illustrating 
distinctions within and between the six EFLs. Three commenters stated that they would like to 
see the proposed six EFLs divided into more than six levels. Finally, one commenter stated that 
the language might be too academic for non-educator audiences and suggested that we alter the 
language to be more accessible to multiple non-educator audiences. 


NRS Implementation Guidelines D-4 


Appendix D: Comments and Responses on Proposed EFL Descriptors for ABE 


Discussion: 


We appreciate the concern these commenters shared regarding the need for clarity of the 
proposed EFL descriptors. We agree that, as a foundational piece of the NRS, it is important that 
adult educators understand each descriptor as well as the interconnectedness among them. 
However, we do not agree that expanding the language of the proposed EFL descriptors, as 
suggested by some commenters, is the best way to achieve that goal. We believe that coordinated 
training and technical assistance—rather than more expansive language for the proposed EFL 
descriptors —can better achieve this aim. In terms of taking a more plain language approach to 
the wording of the proposed revised EFL descriptors, we believe that the language that we use 
should correspond to the language of the intended primary audience as well as the purpose for 
the EFLs within the overall NRS structure. Since the primary purpose of the EFLs is to assist 
educators in documenting mastery of basic skills for performance accountability reporting in an 
education program, we believe that moderate use of academic and technical language is 
appropriate. We are grateful to commenters for the suggestions that they offered as we believe 
these suggestions can help us better design and implement training and technical assistance 
activities as we transition to the proposed EFL descriptors. 


Changes to Proposed EFL Descriptors: 


Considering the above comments and keeping in mind an intended primary audience of 
educators, we made three types of minor revisions to the proposed EFL descriptors for 
English/Language Arts. First, we added ranges of Lexile Measures to each Reading section. 
Second, for Level 1 and 2 (Beginning Literacy and Beginning Basic, respectively) reading, we 
clarified phrasing to more clearly describe the skills and abilities to be demonstrated by students 
exiting these EFLs. For example, in Level 1 Beginning Basic reading we proposed, “They 
[students] are able to distinguish shades of meaning among verbs (e.g., look, glance, stare, glare) 
and adjectives differing in intensity (e.g., large, gigantic).” We revised this sentence to clarify 
how students demonstrate such shades of meaning by stating, “They are able to distinguish 
shades of meaning among verbs (e.g., look, glance, stare, glare) and adjectives differing in 
intensity (e.g., large, gigantic) by choosing them or acting out their meanings [emphasis added]. 
Third, in each of the six Speaking/Listening sections of the proposed EFLs, we similarly clarified 
some of the phrasing to more clearly describe the skills and abilities to be demonstrated by 
students exiting each of the proposed EFLs. 


Comment: 


One commenter suggested that particular portions of Level 1 Beginning Literacy Reading might 
be better placed under Speaking and Listening. The commenter stated that demonstrating 
understanding of spoken words and producing rhyming words appeared more related to spoken 
language than to reading. 


Discussion: 
We appreciate the commenter’s concern for having the proposed EFL descriptors being as clear 


as possible. We understand how these examples taken out of context might be read as examples 
of spoken language skills. However, in the context of reading at the Level 1 Basic Literacy 
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educational functioning level, we proposed them as skills and abilities indicative of the 
development of phonemic awareness which research has demonstrated to be a clear predictor of 
literacy development. In that regard, we do consider them to be appropriately placed based upon 
the current evidence base on reading and literacy development. 


Changes to Proposed EFL Descriptors: 


None 
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New Educational Functioning Level Descriptors for 
Adult Basic Education (ABE)' 


Literacy/English Language Arts 


Introduction 


The Educational Functional Level (EFL) Descriptors for Literacy/English Language Arts are 
intended to guide both teaching and assessment for adult learners. They are divided into six 
EFLs: Beginning Literacy; Beginning Basic; Low Intermediate; High Intermediate; Low Adult 
Secondary; and High Adult Secondary. The descriptors do not provide a complete or 
comprehensive delineation of all of the skills at any given level but provide examples of the most 
critical concepts and skills for the level. 


While these narrative descriptors address the most critical concepts for assessment and 
instruction for adult learners, lesson plans and test items should be based on additional critical 
concepts from state instructional frameworks and standards, as appropriate for the learner and 
state requirements. 


The EFLs for Literacy/English Language arts are organized into reading, writing, speaking and 
listening, and language domains. Emphasis was placed on reading and writing because most 
instruction and assessment attention will be paid to these domains for ABE students. In addition, 
the descriptors were further informed by OCTAE’s Framework for Employability Skills to 
ensure the levels paid adequate attention to workforce preparation. 


Reading 


The reading sections of the descriptors are consistently more comprehensive than the other 
domains. Reading is a critical area for college and career readiness. One of the elements in the 
reading descriptors that draws clear distinctions between competencies required at each level is 
the complexity of the text that students are to be reading. The EFLs specify a staircase of 
increasing text complexity for students to master from beginning basic reading through the 
college and career readiness level. The comprehension skills of reading are to be applied to level- 
appropriate complex text. The reading domain elements of the descriptors carry within it 
references to other key skills from the other domains and workforce preparation skills. Examples 
of this include listening comprehension as a supplement to reading comprehension at levels 1 
and 2 so students can work with the richer ideas adult student can handle intellectually, if not yet 
independently through their own reading. It also includes integrating and evaluating information 
from a variety of media, including translating quantitative or technical information presented 
visually or in words. Learning to work with diverse media is an important job skill as well as a 
critical applied academic skill. Another example is an emphasis on research that includes a 


' These descriptors will not be implemented until the Secretary has determined that there is at least one assessment 
that is both aligned with the revised descriptors and that is suitable for use in the NRS. Until that time, programs 
will continue to use the existing NRS educational functioning level descriptors. 
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combination of reading, writing, and speaking and listening skills—again as a way to connect the 
domains in important ways and to create the EFLs as a focused and useful document. 


Writing 


Details about the level of writing proficiency required at each level have been pared to draw 
clear distinctions between competencies required at each level. The descriptors emphasize 
writing arguments and writing to inform and explain from Level 3 and beyond. Both writing 
types stress writing to sources, and asking students to draw evidence from texts is emphasized in 
the descriptors. With writing, many of the process standards were not included because process 
proficiency is hard to measure. In addition, reference is consistently made to research skills in 
both the reading and writing sections of each level, as these skills are important to writing. 


Speaking and Listening 


The speaking and listening descriptors at each level ware connected closely to workforce 
preparation and the Employability Skills Framework. These skills have the benefit of both being 
measurable and clearly related to citizenship, work and life success. Collaborative conversations 
and teamwork are emphasized at every level, as is students' use of evidence. In this context of 
speaking and listening, the descriptors reflect use of listening comprehension capacities 
(particularly in Levels 1 and 2 to augment students’ lower reading comprehension abilities), 
evidence in conversation, ability to evaluate what others are saying, and the capacity to share 
information effectively with others. 


Language 


In the language domain, descriptors consistent with workforce preparation from the 
Employability Skills Framework and are vital to attaining college and career readiness from each 
level such as a growth in students’ grammar and punctuation skills, as well as their growth in 
vocabulary. 


Level 1: Beginning Literacy 


Reading: Individuals ready to exit the Beginning Literacy Level comprehend how print corresponds 
to spoken language and are able to demonstrate understanding of spoken words, syllables, and 
sound-letter relationships (phonetic patterns), including consonant digraphs and blends. In particular, 
students at this level are able to recognize and produce rhyming words, blend and segment onsets 
and rhymes, isolate and pronounce initial, medial, and final sounds, add or substitute individual 
sounds, and blend and segment single syllable words. They are able to decode two-syllable words 
following basic patterns as well as recognize common high frequency words by sight. Individuals are 
able to read simple decodable texts with accuracy, appropriate rate, and expression. They are able to 
determine the meaning of words and phrases in texts with clear and explicit context. 


Individuals ready to exit this level are able to determine main ideas, retell key details, and ask and 
answer questions about key details in simple texts. Individuals are also able to use the illustrations in 
the text(s), whether print or digital, to describe its key ideas (e.g., maps, charts, photographs, 
cartoons). They also are able to use text features, both print and digital, to locate key facts or 
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information. When listening to text above their current independent reading level, they are able to 
identify the reasons an author gives to support points in a text, describe the connections between 
ideas within a text, and examine the basic similarities in and differences between two texts on the 
same topic. 


Writing: Individuals ready to exit the Beginning Literacy Level are able to write basic sight words 
and familiar words and phrases as they compose simple sentences or phrases. This includes writing 
simple informative texts in which they supply some facts about a topic and narratives that include 
some details regarding what happened. They use simple transition and temporal words to signal 
event order (e.g., so, and, because, when, next, finally). With support, they are able to gather and use 
information from provided sources, both print and digital, to answer a simple research question. 


Speaking and Listening: Individuals ready to exit this level are able to participate in conversations 
of short duration, collaborating with diverse partners and groups, while respecting individual 
differences. This includes following agreed upon rules for discussion and responding to the 
comments of others through multiple exchanges. Individuals are able to describe people, places, 
things, and events with relevant details, producing complete sentences when appropriate to task and 
situation. They can discuss what they have heard read aloud and ask and answer questions about it. 


Language: When writing and speaking, individuals ready to exit this level are able to correctly use 
frequently occurring nouns, verbs (past, present, and future), adjectives, pronouns, prepositions and 
conjunctions. When writing sentences individuals correctly use capitalization, ending punctuation, 
and commas in dates and to separate single words in a series. They are able to spell words with 
common patterns and frequently occurring irregular words. Other words they spell phonetically. In 
response to prompts, they are able to produce and expand complete simple and compound 
declarative, interrogative, imperative, and exclamatory sentences orally. Individuals are able to 
determine the meaning of unknown and multiple-meaning words, by applying their knowledge of 
frequently occurring roots and affixes, as well as sentence-level context. They are able to distinguish 
shades of meaning among verbs (e.g., look, glance, stare, glare) and adjectives differing in intensity 
(e.g., large, gigantic) by choosing them or acting out their meanings. 


Level 2: Beginning Basic 


Reading: Individuals ready to exit the Beginning Basic Level are able to decode multi-syllable 
words, distinguish long and short vowels when reading regularly spelled one-syllable words, and 
recognize the spelling-sound correspondences for common vowel teams. They also are able to 
identify and understand the meaning of the most common prefixes and suffixes. They can read 
common irregular sight words. Individuals are able to read level appropriate texts (e.g., texts with a 
Lexile Measure of between 420 — 820) with accuracy, appropriate rate, and expression.” They are 
able to determine the meaning of words and phrases in level-appropriate complex texts. Individuals 
ready to exit this level are able to determine main ideas, ask and answer questions about key details 
in texts and show how those details support the main idea. Individuals also are able to explain how 
specific aspects of both digital and print illustrations contribute to what is conveyed by the words of 
a text. They are able to compare and contrast the most important points and key details of two texts 


? Refer to the Text Complexity Chart at the end of this document for the CCR standards for adult education for the appropriate range of 
complexity for this level. 
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on the same topic. When listening to text above their current independent reading level, they are able 
to describe the relationship between ideas in a text in terms of time, sequence, and cause/effect, as 
well as use text features and search tools, both print and digital, to locate information relevant to a 
given topic efficiently. They also are able to describe how reasons support specific points an author 
makes in a text and identify the author’s main purpose or what the author wants to answer, explain or 
describe, as well as distinguish their own point of view from that of the author’s. 


Writing: Individuals ready to exit the Beginning Basic Level are able to write opinion pieces on 
topics or texts, supporting a point of view with reasons. They are able to write simple informative 
texts in which they examine a topic and convey information clearly. They also are able to write 
narratives with details that describe actions, thoughts, and feelings. They use transition and temporal 
words (e.g., also, another, more, but) to link ideas and signal event order. Individuals ready to exit 
this level are able to use technology to produce and publish writing as well as to interact and 
collaborate with others. They are able to conduct short research projects and summarize their 
learning in print. This includes taking brief notes from both print and digital sources, and sorting 
evidence into provided categories. 


Speaking and Listening: Individuals ready to exit this level are able to participate in a range of 
collaborative conversations with diverse partners and groups, respecting individual differences. This 
includes gaining the floor in respectful way, linking their comments to the remarks of others, and 
expressing their own ideas, clearly in light of the discussions. Individuals are able to report on a 
topic or text or recount an experience, with appropriate facts, and relevant, descriptive details. They 
are able to speak in complete sentences appropriate to task and situation in order to provide 
requested detail or clarification. They can discuss what they have heard read aloud and provide the 
main ideas and appropriate elaboration and detail about the information presented. 


Language: When writing and speaking, individuals ready to exit this level are able to correctly use 
regular and irregular nouns and verbs, comparative and superlative adjectives and adverbs, and 
coordinating and subordinating conjunctions. When writing simple, compound and complex 
sentences, individuals use correct subject-verb and pronoun-antecedent agreement. They also use 
correct capitalization, ending punctuation, commas, and apostrophes to form contractions and 
possessives. They also are able to spell words with conventional patterns and suffixes. They are able 
to use spelling patterns and generalizations (e.g., word patterns, ending rules) in writing words. In 
response to prompts, they are able to produce, expand, and rearrange simple and compound 
sentences. Individuals are able to determine the meaning of unknown and multiple-meaning words in 
level-appropriate complex texts, including academic words, by applying their knowledge of roots and 
affixes, as well as sentence-level context. They are able to distinguish literal from non-literal 
meaning of words, and shades of meaning among related words that describe states of mind or 
degrees of certainty (e.g., knew, believed, wondered, suspected). They are able to demonstrate 
understanding of and use general academic words that signal spatial and temporal relationships. 


Level 3: Low Intermediate 


Reading: Individuals ready to exit the Low Intermediate Level are able to read fluently text of the 
complexity demanded of this level (e.g., a Lexile Measure of between 740 — 1010).° They are able to 


3 Refer to the Text Complexity Chart at the end of this document for the CCR standards for adult education for the appropriate range of 
complexity for this level. 
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use knowledge of letter-sound correspondences, syllabication patterns, and roots and affixes to 
accurately decode unfamiliar words. They are able to determine the meaning of words and phrases 
(e.g., metaphors and similes) in level-appropriate complex texts. Individuals ready to exit this level 
are able to make logical inferences, summarize central ideas or themes, and explain how they are 
supported by key details. They are able to explain events, procedures, or ideas in historical, scientific, 
or technical texts, including what happened and why. They are able to describe the overall structure 
of a text and compare and contrast the structures of two texts. Individuals ready to exit this level are 
also able to interpret information presented visually, orally or quantitatively to find an answer to a 
question or solve a problem. They display this facility with both print and digital media. Individuals 
are able to explain how authors use reasons and evidence to support particular points in a text and 
can integrate information from several texts, whether print, media, or a mix, on the same topic. They 
are able to describe how point of view influences how events are described. They are able to analyze 
multiple accounts of the same event or topic, noting similarities and differences. They are able to 
produce valid evidence for their findings and assertions. 


Writing: Individuals ready to exit the Low Intermediate Level are able to write opinion pieces on 
topics or texts, supporting a point of view with facts and logically ordered reasons. They are able to 
produce informative texts in which they develop a topic with concrete facts and details. They 
convey information clearly with precise language and well-organized paragraphs. They link ideas, 
opinions and reasons with words, phrases, and clauses (e.g., another, specifically, consequently, 
because). They are also able to use technology (including the Internet) to produce and publish 
writing as well as to interact and collaborate with others. They are able to conduct short research 
projects, making frequent use of on-line as well as print sources. This includes the ability to draw 
evidence from several texts to support an analysis. They are able to summarize or paraphrase 
information from and provide a list of those sources. 


Speaking and Listening: Individuals ready to exit this level are able to participate in a range of 
collaborative conversations with diverse partners and groups, respecting individual differences. This 
includes demonstrating an understanding of teamwork and working well with others by carrying out 
their assigned roles, and posing and responding to specific questions, and making comments that 
contribute to and elaborate on the remarks of others. Individuals are able to report on a topic or text 
or present an opinion, sequencing ideas logically and providing appropriate facts, and relevant, 
descriptive details that support the main ideas or themes. They are able to differentiate between 
contexts that call for formal English and situations where informal discourse is appropriate. They 
also are able to paraphrase and summarize what they have heard aloud and explain how each claim 
is supported by reasons and evidence. 


Language: When writing and speaking, individuals ready to exit this level are able to use verb tenses 
to convey various times, sequences, states, and conditions correctly and recognize inappropriate 
shifts in verb tense. They use prepositions, conjunctions, and interjections properly. Individuals write 
simple, compound and complex sentences and use correct subject-verb and pronoun-antecedent 
agreement throughout a piece of writing. They also use correct capitalization, commas, and 
underlining, quotation marks, and italics to indicate titles of works. They are able to correctly use 
frequently confused words (e.g., to, too, two; there, their) and spell correctly, consulting references as 
needed. They are able to produce complete sentences, recognizing and correcting inappropriate 
fragments and run-ons as well as expand, combine and reduce sentences for meaning, reader interest 
and style. Individuals are able to determine the meaning of unknown and multiple-meaning words in 
level-appropriate complex texts, including academic words, by applying their knowledge of roots and 
affixes, as well as sentence-level context. Individuals are able to interpret figurative language, 
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Including similes and metaphors. They also are able to recognize and explain the meaning of 
common idioms, adages, and proverbs. They are able to demonstrate understanding of and use 
general academic words that signal precise actions or emotions (e.g., whined, stammered), signal 
contrast (e.g., however, nevertheless), or other logical relationships (e.g., however, similarly), and are 
basic to a particular topic (e.g. endangered when discussing animal preservation). 


Level 4: High Intermediate 


Reading: Individuals who are ready to exit the High Intermediate Level are able to read fluently text 
of the complexity demanded of this level (e.g., a Lexile Measure of between 925 — 1185). They 
display increasing facility with academic vocabulary and are able to analyze the impact of a specific 
word choice on meaning and tone in level-appropriate complex texts. 


Individuals are able to make logical inferences by offering several pieces of textual evidence. This 
includes citing evidence to support the analysis of primary and secondary sources in history, as well 
as analysis of science and technical texts. They are able to summarize and analyze central ideas, 
including how they are conveyed through particular details in the text. They also are able to analyze 
how a text makes connections among and distinctions between ideas or events and how major 
sections of a text contribute to the development of the ideas. They also are able to follow multistep 
procedures. Individuals are able to identify aspects of a text that reveal point of view and assess how 
point of view shapes style and content in texts. In addition, they are able to evaluate the validity of 
specific claims an author makes through the sufficiency of the reasoning and evidence supplied in 
the text. This includes analyzing how an author responds to conflicting evidence or viewpoints. They 
are able to analyze how multiple texts address similar themes, including how authors acknowledge 
and respond to conflicting evidence or viewpoints and include or avoid particular facts. Individuals 
are also able to analyze the purpose of information presented in diverse media as well as integrate 
and evaluate content from those sources, including quantitative or technical information presented 
visually and in words. They are able to produce valid evidence for their findings and assertions, 
make sound decisions, and solve problems. 


Writing: Writing in response to one or more text(s), individuals ready to exit this level are able to 
compose arguments and informative texts (this includes the narration of historical events, scientific 
procedures/ experiments, or technical processes). When writing arguments, they are able to 
introduce claims, acknowledge alternate or opposing claims, support claims with clear reasons and 
relevant evidence, and organize them logically in a manner that demonstrates an understanding of 
the topic. When writing informative texts, individuals are able to examine a topic through the 
selection, organization, and analysis of relevant facts, concrete details, quotations and other 
information to aid comprehension. Individuals create cohesion in their writing by clarifying the 
relationships among ideas, reasons, and evidence; using appropriate transitions; and including a 
logical progression of ideas, and maintaining consistency in style and tone. Individuals are able to 
use specific word choices appropriate for the topic, purpose, and audience. They also are able to use 
technology to produce and publish writing and link to and cite sources. They conduct short research 
projects, drawing on several sources. This includes the ability to draw evidence from several texts 
to support an analysis. It also includes the ability to locate and organize information, assess the 


“ Refer to the Text Complexity Chart at the end of this document for the CCR standards for adult education for the appropriate range of 
complexity for this level. 
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credibility and accuracy of each source, and communicate the data and conclusions of others while 
avoiding plagiarism. 


Speaking and Listening: Individuals ready to exit the High Intermediate level collaborate well as a 
member of team by building on others’ ideas, expressing their own clearly and maintaining a 
positive attitude. This includes following the rules for collegial discussions and decision-making 
and tracking progress toward specific goals and deadlines. It also includes the ability to pose 
questions that connect the ideas of several speakers and respond to others’ questions and comments 
with relevant evidence and ideas. During these discussions, individuals are able to qualify, alter, or 
justify their own views in light of the evidence presented by others. Just as in writing, individuals 
are able to delineate a speaker’s argument, evaluating the soundness of the reasoning and relevance 
of the evidence. They are able to identify when irrelevant evidence is introduced. They also are able 
to present their own claims and findings that emphasize salient points in a focused and coherent 
manner, with relevant evidence, valid reasoning, and well-chosen details. Individuals adapt their 
speech to a variety of contexts and tasks, demonstrating a command of formal English when 
indicated or appropriate. 


Language: When writing and speaking, individuals ready to exit the High Intermediate level are 
able to ensure pronouns are in the proper case, recognize and correct inappropriate shifts in pronoun 
number and person, and correct vague or unclear pronouns. They know how to form all verb tenses, 
and recognize and correct inappropriate shifts in verb voice and mood. They know how to 
recognize and correct misplaced and dangling modifiers. They are able to adapt their speech to a 
variety of contexts and tasks when indicated. They are able to choose language that expresses ideas 
precisely and concisely, recognizing and eliminating redundancy and wordiness as well as 
maintaining consistency in style and tone. Though errors may be present, the meaning of their 
written and oral communications is clear. Individuals are able to determine the meaning of 
unknown and multiple-meaning words and phrases as they are used in level—appropriate complex 
texts through context clues, knowledge of affixes and roots, and use of reference materials. 


Level 5: Low Adult Secondary 


Reading: Individuals who are ready to exit Low Adult Secondary Level are able to read fluently texts 
that measure at the secondary level of complexity (e.g., a Lexile Measure of between 1050 — 1335).° 
This includes increasing facility with academic vocabulary and figurative language in level- 
appropriate complex texts. This includes determining the meaning of symbols and key terms used in 
a specific scientific or technical context. They are able to analyze the cumulative impact of specific 
word choices on meaning and tone. Individuals are able to make logical and well-supported 
inferences about those complex texts. They are able to analyze the development of central ideas over 
the course of a text and explain how they are refined by particular sentences, paragraphs, or portions 
of text. They are able to provide an objective summary of a text. They are able to analyze in detail a 
series of events described in text and determine whether earlier events caused later ones or simply 
preceded them. They also are able to follow complex multistep directions or procedures. Individuals 
are able to compare the point of view of two or more authors writing about the same or similar 
topics. They are able to evaluate the validity of specific claims an author makes through the 


> Refer to the Text Complexity Chart at the end of this document for the CCR standards for adult education for the appropriate range of 
complexity for this level. 
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sufficiency and relevance of the reasoning and evidence supplied. They also are able to identify false 
statements and fallacious reasoning. They are able to analyze how multiple texts address related 
themes and concepts, including challenging texts, such as seminal US documents of historical and 
literary significance (e.g., Washington’s Farewell Address, the Gettysburg Address). In addition, 
they are able to contrast the findings presented in a text, noting whether those findings support or 
contradict previous explanations or accounts. Individuals are also able to translate quantitative or 
technical information expressed in words in a text into visual form (e.g., a table or chart) and 
translate information expressed visually or mathematically into words. Through their reading and 
research, they are able to cite strong and thorough textual evidence for their findings and assertions 
to make informed decisions and solve problems. 


Writing: Writing in response to one or more text(s), individuals ready to exit this level are able to 
compose arguments and informative texts (this includes the narration of historical events, scientific 
procedures/ experiments, or technical processes). When writing arguments, they are able to 
introduce precise claims, distinguish the claims from alternate or opposing claims, and support 
claims with clear reasons and relevant and sufficient evidence. When writing informative texts, they 
are able to examine a topic through the effective selection, organization, and analysis of well 
chosen, relevant, and sufficient facts appropriate to the audience’s knowledge of the topic. They use 
appropriate and varied transitions as well as consistency in style and tone to link major sections of 
the text, create cohesion, and establish clear relationships among claims, reasons, and evidence. 
Individuals use precise language and domain-specific vocabulary to manage the complexity of the 
topic. They are also able to take advantage of technology’s capacity to link to other information and 
display information flexibly and dynamically. They conduct short research projects as well as more 
sustained research projects to make informed decisions and solve problems. This includes the ability 
to draw evidence from several texts to support an analysis. It also includes the ability to gather and 
organize information, assess the credibility, accuracy, and usefulness of each source, and 
communicate the data and conclusions of others while avoiding plagiarism. 


Speaking and Listening: Individuals ready to exit the Low Adult Secondary level are able to 
participate in a thoughtful, respectful, and well-reasoned exchange of ideas as a member of a team. 
As they collaborate with peers, they are able to set rules for collegial discussions and decision- 
making, clear goals and deadlines. They are able to propel these conversations forward by 
clarifying, verifying or challenging ideas that are presented, actively incorporating others into the 
discussion, responding thoughtfully to diverse perspectives, and summarizing points of agreement 
and disagreement. They also are able to qualify, alter, or justify their own views and understanding 
in light of the evidence and reasoning presented by others. Just as in writing, individuals are able to 
evaluate a speaker’s point of view, and in particular, assess the links among ideas, word choice, and 
points of emphasis and tone used. They also are able to present their own findings and supporting 
evidence clearly, concisely, and logically such that listeners can follow the line of reasoning. 
Individuals adapt their speech to a variety of contexts and tasks, demonstrating a command of 
formal English when indicated or appropriate. 


Language: Individuals ready to exit the Low Adult Secondary level demonstrate strong control of 
English grammar, usage, and mechanics and use these elements to enhance the presentation of ideas 
both in speech and writing. This includes the use of parallel structure and the correct use of various 
types of phrases and clauses to convey specific meanings. They are able to adapt their speech to a 
variety of contexts and tasks when indicated. Though some errors may be present, meaning of their 
written and oral communications is clear. Individuals are able to determine the meaning of unknown 
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and multiple-meaning words and phrases as they are used in level-appropriate complex texts 
through context clues, knowledge of affixes and roots, and use of reference materials. 


Level 6: High Adult Secondary 


Reading: Individuals who are ready to exit High Adult Secondary Level are able to read fluently at 
the college and career readiness level of text complexity (e.g., a Lexile Measure between 1185 — 
1385).° This includes increasing facility with academic vocabulary and figurative language 
sufficient for reading, writing, speaking, and listening at the college and career readiness level. They 
are able to analyze the cumulative impact of specific word choices on meaning and tone. Individuals 
are able to make logical and well-supported inferences about those complex texts. They are able to 
summarize the challenging ideas, concepts or processes contained within them. They are able to 
paraphrase texts in simpler but still accurate terms. Whether they are conducting analyses of 
complex primary and secondary sources in history or in scientific and technical texts, they are able 
to analyze how the ideas and concepts within them develop and interact. Individuals are able to 
assess how points of view shape style and content in texts with particular attention to distinguishing 
what is directly stated in a text from what is really meant (e.g., satire, sarcasm, irony, or 
understatement). Individuals are able to analyze how multiple texts address related themes and 
concepts, including challenging texts such as US founding documents (Declaration of 
Independence, the Bill of Rights). In addition, they are able to compare and contrast treatments of 
the same topic in several primary and secondary sources. Individuals are also able to integrate and 
evaluate multiple sources of information presented in diverse media in order to address a question. 
Through their reading and research at complex levels, they are able to cite strong and thorough 
textual evidence for their findings and assertions to make sound decisions and solve problems. 


Writing: Writing in response to one or more text(s), individuals ready to exit this level are able to 
compose arguments and informative texts (this includes the narration of historical events, scientific 
procedures/ experiments, or technical processes). When writing arguments, they are able to create 
an organization that establishes clear relationships among the claim(s), counterclaim(s), reasons and 
evidence. They fully develop claims and counterclaims, supplying evidence for each while pointing 
out the strengths and limitations of both in a manner that anticipates the audience’s knowledge level 
and concerns. When writing informative texts, they are able to organize complex ideas, concepts, 
and information to make important connections and distinctions through the effective selection and 
analysis of content. They use appropriate and varied transitions to clarify the relationships among 
complex ideas, create cohesion, and link major sections of the text. Individuals are able to maintain 
a formal style while they attend to the norms and conventions of the discipline in which they are 
writing. They are also able to take advantage of technology’s capacity to link to other information 
and display information flexibly and dynamically. They conduct short research projects as well as 
more sustained research projects that require the synthesis of multiple complex sources to make 
informed decisions and solve problems. This includes the ability to draw evidence from several texts 
to support an analysis. It also includes the ability to gather and organize information, assess the 
credibility, accuracy, and usefulness of each source in answering the research question, noting any 
discrepancies among the data collected. 


° Refer to the Text Complexity Chart at the end of this document for the CCR standards for adult education for the appropriate range of 
complexity for this level. 
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Speaking and Listening: Individuals ready to exit the High Adult Secondary level demonstrate 
flexibility, integrity, and initiative when collaborating as an effective member of a team. They are 
able to manage their time and other resources wisely in order to contribute to the team’s overarching 
goal(s) and meet the agreed upon deadlines. This includes the ability to exercise leadership, resolve 
conflicts as they arise, and pose and respond to questions that relate the current discussion to 
broader themes or larger ideas. They are able to express alternative views clearly and persuasively, 
verify or challenge others’ ideas and conclusions, and think creatively and critically in light of the 
evidence and reasoning presented. Just as in writing, individuals are able to evaluate a speaker’s 
point of view, stance, premises, evidence, reasoning, rhetoric, and tone. They also are able to 
present their own findings and supporting evidence clearly, concisely, and logically such that 
listeners can follow the line of reasoning, making strategic use of digital media Individuals adapt 
their speech to a variety of contexts and tasks, demonstrating a command of formal English when 
indicated or appropriate. 


Language: Individuals ready to exit the High Adult Secondary level demonstrate strong control of 
English grammar, usage, and mechanics and use these elements to enhance the presentation of ideas 
both in speech and writing. This includes the use of parallel structure and the correct use of various 
types of phrases and clauses to convey specific meanings. They are able to adapt their speech to a 
variety of contexts and tasks when indicated. The meaning of their written and oral communications 
is clear. Individuals are able to determine the meaning of unknown and multiple-meaning words 
and phrases as they are used in level-appropriate complex texts through context clues, knowledge of 
affixes and roots, and use of reference materials. 


Resource: Quantitative Analysis Chart for Determining Text Complexity’ 


Degrees of 

CCR Levels Reading Flesch- The Lexile Reading 
of Learning ATOS Power Kincaid Framework Maturity 
B (Level 2) 2.75 —5.14 42-54 1.98 — 5.34 420 — 820 3.53 — 6.13 
C (Level 3) 4.97 — 7.03 52 — 60 4.51 — 7.73 740 — 1010 5.42 — 7.92 
D (Level 4) 7.00 — 9.98 57 — 67 6.51 — 10.34 925 — 1185 7.04 — 9.57 
E (Level 5) 9.67 — 12.01 62 — 72 8.32 — 12.12 1050-1335 | 8.41-10.81 
E (Level 6) | 11.20-— 14.10 67 — 74 10.34 — 14.2 1185-1385 | 9.57—- 12.00 


’ This chart only identifies text complexity for levels B through E. At level A, students are just learning how to 


read, so it is not appropriate to focus on the complexity of the text until level B. 
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Mathematics 


Introduction and Process 


The Educational Functioning Level (EFL) Descriptors for Mathematics are written using the 
College and Career Readiness Standards for Adult Education (CCR) as the foundation. They are 
intended to guide both teaching and assessment for adult learners. While these narrative 
descriptors address the most critical concepts for adult learners (as defined in the Major Work of 
the Level), there are additional concepts found in the CCR standards that support the major work 
for each level, and that are included in these descriptors. Lesson plans and assessment items for 
adult learners should be based on the full text of the CCR standards for each level, using these 
critical concepts as the foundation for lesson development and assessment. 


The mathematics descriptors are divided into six educational functioning levels. The levels are 
Beginning Literacy (corresponding to Level A of the CCR); Beginning Basic (corresponding to 
Level B of the CCR); Low Intermediate (corresponding to Level C of the CCR); Middle 
Intermediate (corresponding to part of the Level D CCR), High Intermediate (corresponding to 
the remainder of the Level D CCR); and Adult Secondary (corresponding to Level E of the 
CCR). Each of the levels corresponds roughly to two grade levels, in K-12 terms, except for 
Level E, which combines the critical concepts of all of grades 9 through 12. Within each level 
the descriptors are further divided by domain: The Mathematical Practices, Number Sense and 
Operations, Algebraic Thinking, Geometry (and Measurement), and Data Analysis (Statistics 
and Probability). 


The descriptors do not provide a complete or comprehensive delineation of all of the skills at any 
given level but provide examples of the most critical concepts and skills for the level to guide 
assessment and instruction. Assessment of the Mathematical Practice descriptors are best 
performed in the classroom using assessments that could be formative or summative and may be 
informal. It should be noted that mathematics placement decisions should take into account the 
reading level of the adult student. Verbally presented application problems at all mathematics 
levels require a minimum reading level. 


Level 1: Beginning Literacy 


The Mathematical Practices: Students prepared to exit this level are able to decipher a simple 
problem presented in a context and reason about and apply correct units to the results. They can 
visualize a situation using manipulatives or drawings and explain their processes and results 
using mathematical terms and symbols appropriate for the level. They recognize errors in the 
work and reasoning of others. They are able to strategically select and use appropriate tools to 
aid in their work, such as pencil/paper, measuring devices, and/or manipulatives. They can see 
patterns and structure in sets of numbers and geometric shapes and use those insights to work 
more efficiently. 


Number Sense and Operations: Students prepared to exit this level have an understanding of 
whole number place value for tens and ones and are able to use their understanding of place 
value to compare two-digit numbers. They are able to add whole numbers within 100 and 
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explain their reasoning, e.g., using concrete models or drawings and strategies based on place 
value and/or properties of operations. They are able to apply their knowledge of whole number 
addition and subtraction to represent and solve word problems that call for addition of three 
whole numbers whose sum is less than 20 by using such problem-solving tools as objects, 
drawings, and/or simple equations. 


Algebraic Thinking: Students prepared to exit this level understand and apply the properties of 
operations to addition and subtraction problems. They understand the relationship between the 
two operations and can determine the unknown number in addition or subtraction equations. 


Geometry and Measurement: Students prepared to exit this level can analyze and compare 2- 
dimensional and 3-dimensional shapes based on their attributes, such as their shape, size, 
orientation, the number of sides and/or vertices (angles), or the lengths of their sides. They can 
reason with two-dimensional shapes (e.g., quadrilaterals and half- and quarter-circles) and with 
three-dimensional shapes (e.g., right prisms, cones, and cylinders) to create composite shapes. 
They are able to measure the length of an object as a whole number of units, which are not 
necessarily standard units, for example measuring the length of a pencil using a paper clip as the 
length unit. 


Data Analysis: Students prepared to exit this level are able to organize, represent, and interpret 
simple data sets (e.g., lists of numbers, shapes, or items) using up to three categories. They can 
answer basic questions related to the total number of data points in a set and the number of data 
points in each category, and can compare the number of data points in the different categories. 


Level 2: Beginning Basic 


The Mathematical Practices: Students prepared to exit this level are able to decipher two-step 
problems presented in a context, visualizing a situation using diagrams or sketches, and 
reasoning about and applying the correct units and the proper degree of precision to the results. 
They can explain their processes and results using mathematical terms and symbols appropriate 
for the level and recognize errors in the reasoning of others. They strategically select and use the 
appropriate tools to aid in their work, such as pencil/paper, measuring devices, manipulatives, 
and/or calculators. They are able to see patterns and structure in sets of numbers, including in 
multiplication or addition tables, and use those insights to work more efficiently. 


Number Sense and Operations: Students prepared to exit this level understand place value for 
whole numbers to 1000 and can use that understanding to read, write, count, compare, and round 
three-digit whole numbers to the nearest 10 or 100. They are able to compute fluently with all 
four operations with whole numbers within 100. They use place value and properties of 
operations to explain why addition and subtraction strategies work, and can demonstrate an 
understanding of the inverse relationship between multiplication and division. They can solve 
one- and two-step word problems involving all four operations within 100 and identify and 
explain arithmetic patterns. They have an understanding of fractions, especially unit fractions, 
and can represent simple fractions on a number line. They understand and can explain 
equivalence of fractions, can recognize and generate simple equivalent fractions, and can 
compare two fractions with the same numerator or denominator by reasoning about their size. 
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Algebraic Thinking: Students prepared to exit this level apply the properties of operations to 
multiplication and division of whole numbers. They understand the relationship between 
multiplication and division and can determine the unknown number in multiplication or division 
equations. 


Geometry and Measurement: Students prepared to exit this level are able to reason about 
geometric shapes and their attributes. They can demonstrate an understanding that different 
shapes might share common attributes (e.g., four sides) and can compare and classify two- 
dimensional shapes, particularly quadrilaterals. They are able to partition shapes into parts with 
equal areas and express the area of each part as a unit fraction of the whole. They can use 
common U.S. Customary and metric units for linear measurements (e.g., inches, feet, 
centimeters, and meters) and solve problems involving measurement and estimation of intervals 
of time, liquid volumes, and masses of objects. They understand the concept of area and can 
relate it to addition and multiplication to solve real-world problems. They also understand, and 
can solve, real-world and mathematical problems involving perimeter of polygons. 


Data Analysis: Students prepared to exit this level are able to draw and interpret simple graphs 
(e.g., bar graphs, picture graphs, and number line diagrams) including scaled bar and picture 
graphs. They can solve one- and two-step problems using scaled bar graphs. They can generate 
measurement data by measuring lengths to the nearest half- and quarter-inch and display that 
data by making a line plot marked off in appropriate units. 


Level 3: Low Intermediate 


The Mathematical Practices: Students prepared to exit this level are able to decipher multi-step 
problems presented in a context and reason about and apply the correct units and the proper 
degree of precision to the results. They can visualize a situation using diagrams or sketches, see 
multiple strategies for solving a problem, explain their processes and results, and recognize 
errors in the work and reasoning of others. They can express themselves using mathematical 
terms and notation appropriate for the level and can strategically select and use tools to aid in 
their work, such as pencil/paper, measuring devices, and/or technology. They are able to see 
patterns and structure in sets of numbers and geometric shapes and use those insights to work 
more efficiently. 


Number Sense and Operations: Students prepared to exit this level understand place value for 
both multi-digit whole numbers and decimals to thousandths, and use their understanding to 
read, write, compare, and round decimals. They are able to use their place value understanding 
and properties of operations to fluently perform operations with multi-digit whole numbers and 
decimals. They can find common factors, common multiples, and understand fraction concepts, 
including fraction equivalence and comparison. They can add, subtract, multiply and divide with 
fractions and mixed numbers. They are able to solve multi-step word problems posed with 
whole numbers and fractions, using the four operations. They also have an understanding of ratio 
concepts and can use ratio language to describe a relationship between two quantities, including 
the concept of a unit rate associated with a ratio. 
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Algebraic Thinking: Students prepared to exit this level are able to apply and extend their 
understanding of arithmetic to algebraic expressions, using a symbol to represent an unknown 
value. They can write, evaluate, and interpret expressions and equations, including expressions 
that arise from formulas used in real-world problems. They can solve real-world and 
mathematical problems by writing and solving simple one-variable equations and write a simple 
inequality that represents a constraint or condition in a real-world or mathematical problem. 
They can represent and analyze quantitative relationships between dependent and independent 
variables. 


Geometry and Measurement: Students prepared to exit this level have a basic understanding of 
the coordinate plane and can plot points (i.e., ordered pairs) and place polygons in the coordinate 
plane to solve real-world and mathematical problems. They can classify two-dimensional shapes 
and use formulas to determine the area of two-dimensional shapes such as triangles and 
quadrilaterals. They can determine the surface area of three-dimensional shapes composed of 
rectangles and triangles, and find the volume of right rectangular prisms. They are able to 
convert like measurement units within a given measurement system (e.g., convert 5 cm to 0.05 
m) and use these conversions to solve multi-step, real-world problems. They are also able to 
solve measurement word problems (such as those that involve area, perimeter, distance, time 
intervals, liquid volumes, mass, and money) that involve simple fractions or decimals. 


Data Analysis and Statistics: Students prepared to exit this level have a basic conceptual 
understanding of statistical variability, including such concepts as center, spread, and the overall 
shape of a distribution of data. They can present data using displays such as dot plots, 
histograms, and box plots. 


Level 4: Middle Intermediate 


The Mathematical Practices: Students prepared to exit this level are able to think critically, 
determine an efficient strategy (from among multiple possible strategies) for solving a multi-step 
problem, and persevere in solving challenging problems. They can express themselves using the 
mathematical terms and notation appropriate to the level. They are able to defend their findings 
and critique the reasoning of others. They are accurate in their calculations and use estimation 
strategies to assess the reasonableness of their results. They can create algebraic and geometric 
models and use them to answer questions and solve problems. They can strategically select and 
use tools to aid in their work, such as pencil/paper, measuring devices, calculators, and/or 
spreadsheets. They are able to see patterns and structure in number sets, data, expressions and 
equations, and geometric figures. 


Number Sense and Operations: Students prepared to exit this level have an understanding of the 
rational number system, including how rational numbers can be represented on a number line 
and pairs of rational numbers can be represented on a coordinate plane. They can apply the 
concept of absolute value to find horizontal and vertical distances. They are able to apply the 
properties of integer exponents and evaluate, estimate, and compare simple square roots and 
cube roots. Individuals at this level also understand ratio, rate, and percent concepts, as well as 
proportional relationships. 
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Algebraic Thinking: Students prepared to exit this level understand the connections between 
proportional relationships, lines, and linear equations. They understand numerical and algebraic 
expressions, and equations and are able to use them to solve real-world and mathematical 
problems. They are able to analyze and solve linear equations and pairs of simultaneous linear 
equations. Individuals at this level are able to define, interpret, and compare linear functions. 


Geometry: Students prepared to exit this level can solve real-world and mathematical problems 
that involve angle measure, circumference, and area of 2-dimensional figures. They are able to 
solve problems involving scale drawings of 2-dimensional geometric figures. They understand 
the concepts of congruence and similarity with respect to 2-dimensional figures. They 
understand the Pythagorean theorem and can apply it to determine missing lengths in right 
triangles. 


Statistics and Probability: Students prepared to exit this level can summarize and describe 
numerical data sets in relation to their context, including determining measures of center and 
variability and describing patterns and/or striking deviations from patterns. They understand and 
can apply the concept of chance, or probability. They are able to use scatter plots for bivariate 
measurement data to describe patterns of association between two quantities (such as clustering, 
outliers, positive or negative association, linear or non-linear association). 


Level 5: High Intermediate 


The Mathematical Practices: Students prepared to exit this level are able to think critically, 
determine an efficient strategy (from among multiple possible strategies) for solving a multi-step 
problem, and persevere in solving challenging problems. They can reason quantitatively, 
including using units as a way to solve problems. They are able to defend their findings and 
critique the reasoning of others. They are accurate in their calculations and use estimation 
strategies to assess the reasonableness of their results. They can create algebraic and geometric 
models and use them to answer questions and solve problems. They can strategically select and 
use tools to aid in their work, such as graphing calculators, spreadsheets, and/or computer 
software. They are able to make generalizations based on patterns and structure they discover in 
number sets, data, expressions and equations, and geometric figures and use these insights to 
work more efficiently. 


Number Sense and Operations: Students prepared to exit this level can reason about and solve 
real-world and mathematical problems that involve the four operations with rational numbers. 
They can apply the concept of absolute value to demonstrate on a number line their 
understanding of addition and subtraction with negative and positive rational numbers. 
Individuals at this level can apply ratio and percent concepts, including using rates and 
proportional relationships to solve multi-step real-world and mathematical problems. 


Algebraic Thinking: Students prepared to exit this level are able to use algebraic and graphical 
representations to solve real-world and mathematical problems, involving linear equations, 
inequalities, and pairs of simultaneous linear equations. Individuals at this level are able to use 
linear functions to describe, analyze, and model linear relationships between quantities. 
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Geometry: Students prepared to exit this level can solve real-world and mathematical problems 
that involve volume and surface area of 3-dimensional geometric figures. They can use informal 
arguments to establish facts about various angle relationships such as the relationships between 
angles created when parallel lines are cut by a transversal. They apply the Pythagorean theorem 
to determine lengths in real-world contexts and distances in the coordinate plane. 


Statistics and Probability: Students prepared to exit this level can use random sampling to draw 
inferences about a population and are able to draw informal comparative inferences about two 
populations using measures of center and measures of variability for numerical data from 
random samples. They can develop, use, and evaluate probability models. They are able to use 
scatter plots for bivariate measurement data to interpret patterns of association between two 
quantities (such as clustering, outliers, positive or negative association, linear or non-linear 
association) and a 2-way table to summarize and interpret bivariate categorical data. 


Level 6: Adult Secondary 


The Mathematical Practices: Students prepared to exit this level are able to think critically, make 
assumptions based on a situation, select an efficient strategy from multiple possible problem- 
solving strategies, plan a solution pathway, and make adjustments as needed when solving 
problems. They persevere in solving challenging problems, including considering analogous, 
simpler problems as a way to solving a more complex one. They can reason quantitatively, 
including through the use of units, and can express themselves using the precise definitions and 
mathematical terms and notation appropriate to the level. They are accurate in their calculations, 
use an appropriate level of precision in finding solutions and reporting results, and use estimation 
strategies to assess the reasonableness of their results. They are able to make conjectures, use 
logic to defend their conclusions, and can detect faulty thinking and errors caused by improper 
use of technology. They can create algebraic and geometric models and use them to answer 
questions, interpret data, make predictions, and solve problems. They can create algebraic and 
geometric models and use them to answer questions, interpret data, make predictions, and solve 
problems. They can strategically select and use tools, such as measuring devices, calculators, 
spreadsheets, and/or computer software, to aid in their work. They are able to see patterns and 
structure in calculations, expressions, and equations and make connections to algebraic 
generalizations, which they use to work more efficiently. 


Number Sense and Operations: Students prepared to exit this level have extended their number 
sense to include irrational numbers, radicals, and rational exponents and understand and use the 
set of real numbers. They are able to assess the reasonableness of calculation results based on the 
limitations of technology or given units and quantities and give results with the appropriate 
degree of precision. 


Algebraic Thinking: Students prepared to exit this level understand the structure of expressions 
and can use that structure to rewrite linear, exponential, and quadratic expressions. They can 
add, subtract, and multiply polynomials that involve linear and/or quadratic expressions. They 
are also able to create linear equations and inequalities and quadratic and simple exponential 
equations to represent relationships between quantities and can represent constraints by linear 
equations or inequalities, or by systems of linear equations and/or inequalities. They can interpret 
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the structure of polynomial and rational expressions and use that structure to identify ways to 
rewrite and operate accurately with them. They can add, subtract, and multiply polynomials that 
extend beyond quadratics. They are able to rearrange formulas to highlight a quantity of interest, 
for example rearranging Ohm’s law, V = IR, to highlight resistance R. They are also able to 
create equations and inequalities representing relationships between quantities, including those 
that extend beyond equations or inequalities arising from linear, quadratic, and simple 
exponential functions to include those arising from simple rational functions. They are able to 
use these equations/inequalities to solve problems both algebraically and graphically. They can 
solve linear equations and inequalities; systems of linear equations; quadratic, simple rational, 
and radical equations in one variable; and recognize how and when extraneous solutions may 
arise. 


Students prepared to exit this level also have a basic understanding of functions, can use function 
notation properly, and use such notation to write a function describing a relationship between 
two quantities. They are able to evaluate functions for inputs in their domains and interpret 
linear, quadratic, and exponential functions that arise in applications in terms of the context. 
They are able to construct, graph, compare, and interpret functions (including, but not limited to, 
linear, quadratic, and exponential). They can sketch graphs given a verbal description of the 
relationship and identify and interpret key features of the graphs of functions that arise in 
applications in a context. They are able to select or define a function that appropriately models a 
relationship and to compare properties of two functions each represented in a different way 
(algebraically, graphically, numerically in tables, or by verbal description). 


Geometry: Students prepared to exit this level can solve problems involving similarity and 
congruence criteria for triangles and use volume formulas for cylinders, pyramids, cones, and 
spheres to solve problems. They can apply the concepts of density based on area and volume in 
modeling situations (e.g., persons per square mile, BTU’s per cubic foot). 


Data Analysis and Statistics: Students prepared to exit this level can summarize, represent, and 
interpret data based on two categorical and quantitative variables, including by using frequency 
tables. They can compare data sets by looking at commonalities and differences in shape, center, 
and spread. They can recognize possible associations and trends in data, in particular in linear 
models, and distinguish between correlation and causation. They interpret one- and two-variable 
data, including those with linear and non-linear relationships. They interpret the slope (rate of 
change) and intercept (constant term) for a line of best fit and in the context of the data. They 
understand and account for extreme points of data in their analysis and interpret relative 
frequencies (joint, marginal and conditional). 
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